SlideShare a Scribd company logo
1 of 36
Download to read offline
Kubernetes in Production,
From the Ground Up
Mike Splain | @mikesplain
About Me
• Lead DevOps Engineer at Barkly
• Formerly PayPal / eBay & OurStage
• Running applications at scale with the least amount of human interaction
• Current Technologies: Kubernetes, Docker, Terraform, Ansible
• Previous Technologies: Chef, Test-Kitchen….
Resources
• docs.kubernetes.io
• coreos.com/kubernetes/docs
• bostonkubernetes.com
• slack.kubernetes.io
• github.com/mikesplain/etcd_demo
Problem:
Our requirements for running our apps
• Fast recovery without human intervention
• Nodes are ephemeral
• Autoscaling
• Testable on developer machines
• Infrastructure as an artifact
Technology choices for running K8s
Hasn’t someone else already built this?
… March 2015
K8s official scripts
• Simple bash scripts
• “Just Works!”
• Sets up Autoscaling group for
minions
• Uses Salt
Problems? (at the time)
• No etcd HA
• No Master HA
• Salt Master is coupled with K8s
Master
• Ubuntu / Fedora
CoreOS’s official scripts
• Cool go app to start it!
• Or Cloud formation?
• But what now?
Problems? (at the time)
• No etcd HA
• No Master HA
• Lots of magic
“It can’t be that hard right?”
etcd
Note to self… you’re going too slow.
etcd
• Easy.
• “Just works”
• Cluster discovery:
• Discovery Service
• DNS
• ?
#cloud-config
coreos:
etcd2:
advertise-client-urls: "http://$public_ipv4:2379"
initial-advertise-peer-urls: "http://$private_ipv4:2380"
listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001"
listen-peer-urls: "http://$private_ipv4:2380,http://$private_ipv4:7001"
discovery-token: “<token here>”
units:
- name: etcd2.service
command: start
update:
reboot-strategy: none
etcd
• etcd-aws-cluster

https://github.com/MonsantoCo/
etcd-aws-cluster
• Uses Autoscaling groups for
discovery
• Requires IAM Instance Roles
#cloud-config
coreos:
etcd2:
advertise-client-urls: "http://$public_ipv4:2379"
initial-advertise-peer-urls: "http://$private_ipv4:2380"
listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001"
listen-peer-urls: "http://$private_ipv4:2380,http://$private_ipv4:7001"
units:
- name: etcd2.service
command: stop
- name: etcd-peers.service
command: start
content: |
[Unit]
Description=Write a file with the etcd peers that we should
bootstrap to
Requires=docker.service
After=docker.service
[Service]
Restart=on-failure
RestartSec=10
TimeoutStartSec=300
ExecStartPre=/usr/bin/docker pull registry.barklyprotects.com/
kubernetes/etcd-aws-cluster:latest
ExecStartPre=/usr/bin/docker run --rm=true -v /etc/sysconfig/:/etc/
sysconfig/ registry.barklyprotects.com/kubernetes/etcd-aws-cluster:latest
ExecStart=/usr/bin/systemctl start etcd2
write_files:
- path: /etc/systemd/system/etcd2.service.d/30-etcd_peers.conf
permissions: 0644
content: |
[Service]
# Load the other hosts in the etcd leader autoscaling group from file
EnvironmentFile=/etc/sysconfig/etcd-peers
Terraform to launch etcd
• References static cloud-init files
resource "aws_launch_configuration" "terraform_etcd" {
name_prefix = "${var.environment}_etcd_conf-"
image_id = "${var.coreos_ami}"
instance_type = "t2.small"
key_name = "${var.key_name}"
security_groups = ["$
{aws_security_group.terraform_etcd2_sec_group.id}"]
user_data = "${file("../cloud-config/output/etcd.yml")}"
enable_monitoring = true
ebs_optimized = false
iam_instance_profile = "$
{aws_iam_instance_profile.terraform_etcd_role_profile.id}"
root_block_device {
volume_size = 20
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_autoscaling_group" "terraform_etcd" {
name = "${var.environment}_etcd"
launch_configuration = "$
{aws_launch_configuration.terraform_etcd.name}"
availability_zones = ["us-east-1c"]
max_size = "${var.capacities_etcd_max}"
min_size = "${var.capacities_etcd_min}"
health_check_grace_period = 300
desired_capacity = "${var.capacities_etcd_desired}"
vpc_zone_identifier = ["${aws_subnet.etcd.id}"]
force_delete = true
tag {
key = "Name"
value = "${var.environment}_etcd"
propagate_at_launch = true
}
lifecycle {
create_before_destroy = true
}
}
Masters
Master Node
• Kubelet service
• API Server
• Controller Manager
• Scheduler
• Proxy
• Flannel
Consider Master pods as
additional artifact
• Docker container that takes env
variables
• Outputs templated pods to disk for
Kubelet to load
• We use j2cli to template these files
with little overhead
apiVersion: v1
kind: Pod
metadata:
name: kube-podmaster
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: scheduler-elector
image: gcr.io/google_containers/podmaster:1.1
imagePullPolicy: Always
command:
- /podmaster
- --etcd-servers={{ ETCD_ENDPOINTS }}
- --key=scheduler
- --whoami={{ ADVERTISE_IP }}
- --source-file=/src/manifests/kube-scheduler.yaml
- --dest-file=/dst/manifests/kube-scheduler.yaml
volumeMounts:
- mountPath: /src/manifests
name: manifest-src
readOnly: true
- mountPath: /dst/manifests
name: manifest-dst
Master Cloud Config
• Setup box for services needed for
real Docker to run
• Get etcd server IPs and write to file
• Start flannel with those IPs
#cloud-config
coreos:
units:
- name: etcd.service
command: stop
- name: etcd2.service
command: stop
- name: early-docker.service
command: start
- name: kub_get_etcd.service
command: start
content: |
[Unit]
Description= Write K8s etcd urls to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c "/usr/bin/docker pull
registry.barklyprotects.com/kubernetes/kub-get-etcd"
ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/
barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{
KUB_ETCD_ASG }} > /etc/etcd_servers.env"
- name: flanneld.service
command: start
drop-ins:
- name: 10-environment_vars.conf
content: |
[Unit]
After=kub_get_etcd.service
[Service]
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n
FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env
>> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE=
$private_ipv4 >> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/
flannel/options.env
Restart=always
RestartSec=10
Other config
• Grab certs from S3
• Terraform only allows
permissions to specific files
• Format Master pod files to disk
- name: kub_certs.service
command: start
content: |
[Unit]
Description=Writes kubernetes cluster certs to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
Before=kubelet.service
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c /usr/bin/mkdir -p /etc/kubernetes/ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/ca.pem /ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/apiserver.pem /ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/apiserver-key.pem /ssl
- name: kub_pods.service
command: start
content: |
[Unit]
Description=Writes kubernetes pod files to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
Before=kubelet.service
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c "/usr/bin/mkdir -p /etc/kubernetes/ssl"
ExecStart=/usr/bin/docker run --net=host -v /etc/barkly/:/etc/
barkly/ -e K8S_VERSION='1.1.7' -e CLOUD_PROVIDER='--cloud-provider=aws' -e
SERVICE_IP_RANGE="10.3.0.0/16" -e ADVERTISE_IP="$private_ipv4" -e
ETCD_AUTOSCALE_GROUP_NAME="our_etcd_autoscaling_group_name" -v /srv/
kubernetes/manifests:/output_src -v /etc/kubernetes/manifests:/output_dst
registry.barklyprotects.com/kubernetes/kub-master-pods
Start Docker & Kubelet
• Kubelet will wait for docker and
flannel to be ready
• Kubelet will load manifests for
Master services outputted from
previous container
- name: docker.service
command: start
drop-ins:
- name: 40-flannel.conf
content: |
[Unit]
Requires=flanneld.service
After=flanneld.service
- name: kubelet.service
command: start
content: |
[Unit]
Requires=docker.service
After=docker.service
After=fluentd-elasticsearch.service
[Service]
ExecStartPre=/usr/bin/mkdir -p /var/log/containers
ExecStart=/etc/bin/kubelet 
--hostname-override="$private_ipv4" 
--api_servers=http://127.0.0.1:8080 
--register-node=false 
--allow-privileged=true 
--config=/etc/kubernetes/manifests 
--cluster-dns=10.3.0.10 
--cluster-domain=cluster.local 
--cloud-provider=aws 
--v=4
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
Get Kubelet
• /usr/bin & /usr/local/bin are read
only in CoreOS
- name: kubelet.service
command: start
drop-ins:
- name: 10-download-binary.conf
content: |
[Service]
ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet"
write_files:
# Since systemd needs these files before it will start
- path: /etc/bin/download-k8s-binary
permissions: '0755'
content: |
#!/usr/bin/env bash
export K8S_VERSION=“v1.2.4"
mkdir -p /etc/bin
FILE=$1
if [ ! -f /usr/bin/$FILE ]; then
curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly-
kubernetes-builds/${K8S_VERSION}/bin/$FILE
chmod +x /etc/bin/$FILE
else
# we check the version of the binary
INSTALLED_VERSION=$(/etc/bin/$FILE --version)
MATCH=$(echo "${INSTALLED_VERSION}" | grep -c "${K8S_VERSION}")
if [ $MATCH -eq 0 ]; then
# the version is different
curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly-
kubernetes-builds/${K8S_VERSION}/bin/$FILE
chmod +x /etc/bin/$FILE
fi
fi
Terraform to build
• Similar to as before.. we reference
our cloudinit script
resource "aws_launch_configuration" "terraform_master" {
name_prefix = "${var.environment}_master_conf-"
image_id = "${var.coreos_ami}"
instance_type = "t2.medium"
key_name = "${var.key_name}"
security_groups = ["$
{aws_security_group.terraform_master_sec_group.id}"]
user_data = "${file("../cloud-config/output/master.yml")}"
enable_monitoring = true
ebs_optimized = false
iam_instance_profile = "$
{aws_iam_instance_profile.terraform_master_role_profile.id}"
root_block_device {
volume_size = 20
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_autoscaling_group" "terraform_master" {
name = "${var.environment}_master"
launch_configuration = "$
{aws_launch_configuration.terraform_master.name}"
availability_zones = ["us-east-1c"]
max_size = "${var.capacities_master_max}"
min_size = "${var.capacities_master_min}"
health_check_grace_period = 300
desired_capacity = "${var.capacities_master_desired}"
vpc_zone_identifier = ["${aws_subnet.master.id}"]
force_delete = true
load_balancers = ["${aws_elb.terraform_master.name}"]
tag {
key = "Name"
value = "${var.environment}_master"
propagate_at_launch = true
}
lifecycle {
create_before_destroy = true
}
}
Minions
Minion Node (now just Nodes)
• Kubelet Service manages all other services.
• Proxy
• Pods
Early Docker / Flannel
• Same as master!
#cloud-config
coreos:
units:
- name: etcd.service
command: stop
- name: etcd2.service
command: stop
- name: early-docker.service
command: start
- name: kub_get_etcd.service
command: start
content: |
[Unit]
Description= Write K8s etcd urls to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c "/usr/bin/docker pull
registry.barklyprotects.com/kubernetes/kub-get-etcd"
ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/
barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{
KUB_ETCD_ASG }} > /etc/etcd_servers.env"
- name: flanneld.service
command: start
drop-ins:
- name: 10-environment_vars.conf
content: |
[Unit]
After=kub_get_etcd.service
[Service]
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n
FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env
>> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE=
$private_ipv4 >> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/
flannel/options.env
Restart=always
RestartSec=10
Kubelet
• Similar to master
- name: docker.service
command: start
drop-ins:
- name: 40-flannel.conf
content: |
[Unit]
Requires=flanneld.service
After=flanneld.service
- name: kubelet.service
command: start
content: |
[Unit]
Requires=docker.service
After=docker.service
After=fluentd-elasticsearch.service
[Service]
ExecStartPre=/usr/bin/mkdir -p /var/log/containers
ExecStart=/etc/bin/kubelet 
--api_servers=https://ourk8smaster.barkly.com 
--hostname-override="$private_ipv4" 
--register-node=true 
--allow-privileged=true 
--config=/etc/kubernetes/manifests 
--cluster-dns=10.3.0.10 
--cluster-domain=cluster.local 
--kubeconfig=/etc/kubernetes/worker-kubeconfig-kubelet.yaml 
--tls-cert-file=/etc/kubernetes/ssl/worker.pem 
--tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem 
--cloud-provider=aws 
--v=4
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
drop-ins:
- name: 10-download-binary.conf
content: |
[Service]
ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet"
Manifests can be hard coded
• Minion manifests are far less
dynamic and can be hard coded
as CloudInit files
write_files:
- path: "/etc/kubernetes/manifests/kube-proxy.yaml"
content: |
apiVersion: v1
kind: Pod
metadata:
name: kube-proxy
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-proxy
image: registry.barklyprotects.com/kubernetes/hyperkube:v1.1.8
command:
- /hyperkube
- proxy
- --master=https://ourk8smaster.barkly.com
- --kubeconfig=/etc/kubernetes/worker-kubeconfig-proxy.yaml
- --proxy-mode=iptables
- --v=4
securityContext:
privileged: true
volumeMounts:
- mountPath: /etc/ssl/certs
name: "ssl-certs"
- mountPath: /etc/kubernetes/worker-kubeconfig-proxy.yaml
name: "kubeconfig"
readOnly: true
- mountPath: /etc/kubernetes/ssl
name: "etc-kube-ssl"
readOnly: true
volumes:
- name: "ssl-certs"
hostPath:
path: "/usr/share/ca-certificates"
- name: "kubeconfig"
hostPath:
path: "/etc/kubernetes/worker-kubeconfig-proxy.yaml"
- name: "etc-kube-ssl"
hostPath:
path: "/etc/kubernetes/ssl"
Manifests can be hard coded
• Minion manifests are far less
dynamic and can be hard coded
as CloudInit files
- path: "/etc/kubernetes/worker-kubeconfig-proxy.yaml"
content: |
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
certificate-authority: /etc/kubernetes/ssl/ca.pem
users:
- name: kubelet
user:
client-certificate: /etc/kubernetes/ssl/worker.pem
client-key: /etc/kubernetes/ssl/worker-key.pem
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
- path: "/etc/kubernetes/worker-kubeconfig-kubelet.yaml"
content: |
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
certificate-authority: /etc/kubernetes/ssl/ca.pem
users:
- name: kubelet
user:
client-certificate: /etc/kubernetes/ssl/worker.pem
client-key: /etc/kubernetes/ssl/worker-key.pem
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
Results
K8s does not so well:
• Still relatively young,
• Not extremely google-able
• Leaves managing manifests to user with little guidance
• Docker only (rkt coming soon)
• No DNS support (coming soon)
• SaltStack by default
K8s does really well:
• Keeping applications running
• Managing failover, liveness,readiness of applications
• Mounting volumes, load balancers (cloud resources)
• Internal DNS, Namespacing
• Basic secret / config management
• Easily Horizontally Scalable services
• Rolling updates / rollback via Deployments
• Support via Github / Slack chat
• Easy debugging of actual app problems via kubectl logs and kubectl exec
Demo!
https://github.com/mikesplain/etcd_demo
Questions?

More Related Content

What's hot

What's hot (20)

DevOps in AWS with Kubernetes
DevOps in AWS with KubernetesDevOps in AWS with Kubernetes
DevOps in AWS with Kubernetes
 
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
 
Tectonic Summit 2016: Kubernetes 1.5 and Beyond
Tectonic Summit 2016: Kubernetes 1.5 and BeyondTectonic Summit 2016: Kubernetes 1.5 and Beyond
Tectonic Summit 2016: Kubernetes 1.5 and Beyond
 
Kubernetes automation in production
Kubernetes automation in productionKubernetes automation in production
Kubernetes automation in production
 
Beyond static configuration
Beyond static configurationBeyond static configuration
Beyond static configuration
 
An Introduction to the Kubernetes API
An Introduction to the Kubernetes APIAn Introduction to the Kubernetes API
An Introduction to the Kubernetes API
 
Achieving CI/CD with Kubernetes
Achieving CI/CD with KubernetesAchieving CI/CD with Kubernetes
Achieving CI/CD with Kubernetes
 
Introduction to Docker
Introduction to DockerIntroduction to Docker
Introduction to Docker
 
Package your Java EE Application using Docker and Kubernetes
Package your Java EE Application using Docker and KubernetesPackage your Java EE Application using Docker and Kubernetes
Package your Java EE Application using Docker and Kubernetes
 
Scaling docker with kubernetes
Scaling docker with kubernetesScaling docker with kubernetes
Scaling docker with kubernetes
 
Kubernetes and Hybrid Deployments
Kubernetes and Hybrid DeploymentsKubernetes and Hybrid Deployments
Kubernetes and Hybrid Deployments
 
Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10
 
Docker serverless v1.0
Docker serverless v1.0Docker serverless v1.0
Docker serverless v1.0
 
Orchestrating Linux Containers while tolerating failures
Orchestrating Linux Containers while tolerating failuresOrchestrating Linux Containers while tolerating failures
Orchestrating Linux Containers while tolerating failures
 
Docker for Fun and Profit
Docker for Fun and ProfitDocker for Fun and Profit
Docker for Fun and Profit
 
Orchestration tool roundup kubernetes vs. docker vs. heat vs. terra form vs...
Orchestration tool roundup   kubernetes vs. docker vs. heat vs. terra form vs...Orchestration tool roundup   kubernetes vs. docker vs. heat vs. terra form vs...
Orchestration tool roundup kubernetes vs. docker vs. heat vs. terra form vs...
 
IPC16: A Practical Introduction to Kubernetes
IPC16: A Practical Introduction to Kubernetes IPC16: A Practical Introduction to Kubernetes
IPC16: A Practical Introduction to Kubernetes
 
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?
 
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2days
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2daysUsing Kubernetes for Continuous Integration and Continuous Delivery. Java2days
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2days
 
Kubernetes Introduction
Kubernetes IntroductionKubernetes Introduction
Kubernetes Introduction
 

Viewers also liked

Viewers also liked (12)

KELK Stack on AWS
KELK Stack on AWSKELK Stack on AWS
KELK Stack on AWS
 
Cloud Solution Day 2016: Service Mesh for Kubernetes
Cloud Solution Day 2016: Service Mesh for KubernetesCloud Solution Day 2016: Service Mesh for Kubernetes
Cloud Solution Day 2016: Service Mesh for Kubernetes
 
Running Production-Grade Kubernetes on AWS
Running Production-Grade Kubernetes on AWSRunning Production-Grade Kubernetes on AWS
Running Production-Grade Kubernetes on AWS
 
Kubernetes on AWS
Kubernetes on AWSKubernetes on AWS
Kubernetes on AWS
 
Webcast - Making kubernetes production ready
Webcast - Making kubernetes production readyWebcast - Making kubernetes production ready
Webcast - Making kubernetes production ready
 
Large Scale Kubernetes on AWS at Europe's Leading Online Fashion Platform - A...
Large Scale Kubernetes on AWS at Europe's Leading Online Fashion Platform - A...Large Scale Kubernetes on AWS at Europe's Leading Online Fashion Platform - A...
Large Scale Kubernetes on AWS at Europe's Leading Online Fashion Platform - A...
 
From dev to prod: Kubernetes on AWS (short ver.)
From dev to prod: Kubernetes on AWS (short ver.)From dev to prod: Kubernetes on AWS (short ver.)
From dev to prod: Kubernetes on AWS (short ver.)
 
Kubernetes Architecture - beyond a black box - Part 1
Kubernetes Architecture - beyond a black box - Part 1Kubernetes Architecture - beyond a black box - Part 1
Kubernetes Architecture - beyond a black box - Part 1
 
Kubernetes on AWS at Europe's Leading Online Fashion Platform
Kubernetes on AWS at Europe's Leading Online Fashion PlatformKubernetes on AWS at Europe's Leading Online Fashion Platform
Kubernetes on AWS at Europe's Leading Online Fashion Platform
 
Kubernetes networking in AWS
Kubernetes networking in AWSKubernetes networking in AWS
Kubernetes networking in AWS
 
O'Reilly Software Architecture Conference London 2017: Building Resilient Mic...
O'Reilly Software Architecture Conference London 2017: Building Resilient Mic...O'Reilly Software Architecture Conference London 2017: Building Resilient Mic...
O'Reilly Software Architecture Conference London 2017: Building Resilient Mic...
 
Beyond Ingresses - Better Traffic Management in Kubernetes
Beyond Ingresses - Better Traffic Management in KubernetesBeyond Ingresses - Better Traffic Management in Kubernetes
Beyond Ingresses - Better Traffic Management in Kubernetes
 

Similar to Container Days Boston - Kubernetes in production

PDXPortland - Dockerize Django
PDXPortland - Dockerize DjangoPDXPortland - Dockerize Django
PDXPortland - Dockerize Django
Hannes Hapke
 

Similar to Container Days Boston - Kubernetes in production (20)

Kubernetes Boston — Custom High Availability of Kubernetes
Kubernetes Boston — Custom High Availability of KubernetesKubernetes Boston — Custom High Availability of Kubernetes
Kubernetes Boston — Custom High Availability of Kubernetes
 
Our Puppet Story (GUUG FFG 2015)
Our Puppet Story (GUUG FFG 2015)Our Puppet Story (GUUG FFG 2015)
Our Puppet Story (GUUG FFG 2015)
 
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
Docker San Francisco Meetup April 2015 - The Docker Orchestration Ecosystem o...
 
Terraform in deployment pipeline
Terraform in deployment pipelineTerraform in deployment pipeline
Terraform in deployment pipeline
 
Dev ops meetup
Dev ops meetupDev ops meetup
Dev ops meetup
 
Agiles Peru 2019 - Infrastructure As Code
Agiles Peru 2019 - Infrastructure As CodeAgiles Peru 2019 - Infrastructure As Code
Agiles Peru 2019 - Infrastructure As Code
 
Iac d.damyanov 4.pptx
Iac d.damyanov 4.pptxIac d.damyanov 4.pptx
Iac d.damyanov 4.pptx
 
Ansible benelux meetup - Amsterdam 27-5-2015
Ansible benelux meetup - Amsterdam 27-5-2015Ansible benelux meetup - Amsterdam 27-5-2015
Ansible benelux meetup - Amsterdam 27-5-2015
 
Can puppet help you run docker on a T2.Micro?
Can puppet help you run docker on a T2.Micro?Can puppet help you run docker on a T2.Micro?
Can puppet help you run docker on a T2.Micro?
 
London HUG 12/4
London HUG 12/4London HUG 12/4
London HUG 12/4
 
Managing Infrastructure as Code
Managing Infrastructure as CodeManaging Infrastructure as Code
Managing Infrastructure as Code
 
Cloud patterns applied
Cloud patterns appliedCloud patterns applied
Cloud patterns applied
 
Infrastructure-as-Code (IaC) Using Terraform (Advanced Edition)
Infrastructure-as-Code (IaC) Using Terraform (Advanced Edition)Infrastructure-as-Code (IaC) Using Terraform (Advanced Edition)
Infrastructure-as-Code (IaC) Using Terraform (Advanced Edition)
 
Terraform Cosmos DB
Terraform Cosmos DBTerraform Cosmos DB
Terraform Cosmos DB
 
"Continuously delivering infrastructure using Terraform and Packer" training ...
"Continuously delivering infrastructure using Terraform and Packer" training ..."Continuously delivering infrastructure using Terraform and Packer" training ...
"Continuously delivering infrastructure using Terraform and Packer" training ...
 
Declarative Infrastructure Tools
Declarative Infrastructure Tools Declarative Infrastructure Tools
Declarative Infrastructure Tools
 
Introduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release updateIntroduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release update
 
Openstack Magnum: Container-as-a-Service
Openstack Magnum: Container-as-a-ServiceOpenstack Magnum: Container-as-a-Service
Openstack Magnum: Container-as-a-Service
 
PDXPortland - Dockerize Django
PDXPortland - Dockerize DjangoPDXPortland - Dockerize Django
PDXPortland - Dockerize Django
 
OSDC 2015: Mitchell Hashimoto | Automating the Modern Datacenter, Development...
OSDC 2015: Mitchell Hashimoto | Automating the Modern Datacenter, Development...OSDC 2015: Mitchell Hashimoto | Automating the Modern Datacenter, Development...
OSDC 2015: Mitchell Hashimoto | Automating the Modern Datacenter, Development...
 

Recently uploaded

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
drm1699
 

Recently uploaded (20)

Rapidoform for Modern Form Building and Insights
Rapidoform for Modern Form Building and InsightsRapidoform for Modern Form Building and Insights
Rapidoform for Modern Form Building and Insights
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test Automation
 
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
Incident handling is a clearly defined set of procedures to manage and respon...
Incident handling is a clearly defined set of procedures to manage and respon...Incident handling is a clearly defined set of procedures to manage and respon...
Incident handling is a clearly defined set of procedures to manage and respon...
 
Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14Spring into AI presented by Dan Vega 5/14
Spring into AI presented by Dan Vega 5/14
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
 

Container Days Boston - Kubernetes in production

  • 1. Kubernetes in Production, From the Ground Up Mike Splain | @mikesplain
  • 2. About Me • Lead DevOps Engineer at Barkly • Formerly PayPal / eBay & OurStage • Running applications at scale with the least amount of human interaction • Current Technologies: Kubernetes, Docker, Terraform, Ansible • Previous Technologies: Chef, Test-Kitchen….
  • 3. Resources • docs.kubernetes.io • coreos.com/kubernetes/docs • bostonkubernetes.com • slack.kubernetes.io • github.com/mikesplain/etcd_demo
  • 5. Our requirements for running our apps • Fast recovery without human intervention • Nodes are ephemeral • Autoscaling • Testable on developer machines • Infrastructure as an artifact
  • 7. Hasn’t someone else already built this? … March 2015
  • 8. K8s official scripts • Simple bash scripts • “Just Works!” • Sets up Autoscaling group for minions • Uses Salt Problems? (at the time) • No etcd HA • No Master HA • Salt Master is coupled with K8s Master • Ubuntu / Fedora
  • 9. CoreOS’s official scripts • Cool go app to start it! • Or Cloud formation? • But what now? Problems? (at the time) • No etcd HA • No Master HA • Lots of magic
  • 10. “It can’t be that hard right?”
  • 11.
  • 12.
  • 13. etcd
  • 14. Note to self… you’re going too slow.
  • 15. etcd • Easy. • “Just works” • Cluster discovery: • Discovery Service • DNS • ? #cloud-config coreos: etcd2: advertise-client-urls: "http://$public_ipv4:2379" initial-advertise-peer-urls: "http://$private_ipv4:2380" listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001" listen-peer-urls: "http://$private_ipv4:2380,http://$private_ipv4:7001" discovery-token: “<token here>” units: - name: etcd2.service command: start update: reboot-strategy: none
  • 16. etcd • etcd-aws-cluster
 https://github.com/MonsantoCo/ etcd-aws-cluster • Uses Autoscaling groups for discovery • Requires IAM Instance Roles #cloud-config coreos: etcd2: advertise-client-urls: "http://$public_ipv4:2379" initial-advertise-peer-urls: "http://$private_ipv4:2380" listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001" listen-peer-urls: "http://$private_ipv4:2380,http://$private_ipv4:7001" units: - name: etcd2.service command: stop - name: etcd-peers.service command: start content: | [Unit] Description=Write a file with the etcd peers that we should bootstrap to Requires=docker.service After=docker.service [Service] Restart=on-failure RestartSec=10 TimeoutStartSec=300 ExecStartPre=/usr/bin/docker pull registry.barklyprotects.com/ kubernetes/etcd-aws-cluster:latest ExecStartPre=/usr/bin/docker run --rm=true -v /etc/sysconfig/:/etc/ sysconfig/ registry.barklyprotects.com/kubernetes/etcd-aws-cluster:latest ExecStart=/usr/bin/systemctl start etcd2 write_files: - path: /etc/systemd/system/etcd2.service.d/30-etcd_peers.conf permissions: 0644 content: | [Service] # Load the other hosts in the etcd leader autoscaling group from file EnvironmentFile=/etc/sysconfig/etcd-peers
  • 17. Terraform to launch etcd • References static cloud-init files resource "aws_launch_configuration" "terraform_etcd" { name_prefix = "${var.environment}_etcd_conf-" image_id = "${var.coreos_ami}" instance_type = "t2.small" key_name = "${var.key_name}" security_groups = ["$ {aws_security_group.terraform_etcd2_sec_group.id}"] user_data = "${file("../cloud-config/output/etcd.yml")}" enable_monitoring = true ebs_optimized = false iam_instance_profile = "$ {aws_iam_instance_profile.terraform_etcd_role_profile.id}" root_block_device { volume_size = 20 } lifecycle { create_before_destroy = true } } resource "aws_autoscaling_group" "terraform_etcd" { name = "${var.environment}_etcd" launch_configuration = "$ {aws_launch_configuration.terraform_etcd.name}" availability_zones = ["us-east-1c"] max_size = "${var.capacities_etcd_max}" min_size = "${var.capacities_etcd_min}" health_check_grace_period = 300 desired_capacity = "${var.capacities_etcd_desired}" vpc_zone_identifier = ["${aws_subnet.etcd.id}"] force_delete = true tag { key = "Name" value = "${var.environment}_etcd" propagate_at_launch = true } lifecycle { create_before_destroy = true } }
  • 19. Master Node • Kubelet service • API Server • Controller Manager • Scheduler • Proxy • Flannel
  • 20. Consider Master pods as additional artifact • Docker container that takes env variables • Outputs templated pods to disk for Kubelet to load • We use j2cli to template these files with little overhead apiVersion: v1 kind: Pod metadata: name: kube-podmaster namespace: kube-system spec: hostNetwork: true containers: - name: scheduler-elector image: gcr.io/google_containers/podmaster:1.1 imagePullPolicy: Always command: - /podmaster - --etcd-servers={{ ETCD_ENDPOINTS }} - --key=scheduler - --whoami={{ ADVERTISE_IP }} - --source-file=/src/manifests/kube-scheduler.yaml - --dest-file=/dst/manifests/kube-scheduler.yaml volumeMounts: - mountPath: /src/manifests name: manifest-src readOnly: true - mountPath: /dst/manifests name: manifest-dst
  • 21. Master Cloud Config • Setup box for services needed for real Docker to run • Get etcd server IPs and write to file • Start flannel with those IPs #cloud-config coreos: units: - name: etcd.service command: stop - name: etcd2.service command: stop - name: early-docker.service command: start - name: kub_get_etcd.service command: start content: | [Unit] Description= Write K8s etcd urls to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c "/usr/bin/docker pull registry.barklyprotects.com/kubernetes/kub-get-etcd" ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/ barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{ KUB_ETCD_ASG }} > /etc/etcd_servers.env" - name: flanneld.service command: start drop-ins: - name: 10-environment_vars.conf content: | [Unit] After=kub_get_etcd.service [Service] ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE= $private_ipv4 >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/ flannel/options.env Restart=always RestartSec=10
  • 22. Other config • Grab certs from S3 • Terraform only allows permissions to specific files • Format Master pod files to disk - name: kub_certs.service command: start content: | [Unit] Description=Writes kubernetes cluster certs to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target Before=kubelet.service [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c /usr/bin/mkdir -p /etc/kubernetes/ssl ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/ ssl registry.barklyprotects.com/ops/awscli s3 cp s3:// our_k8s_cluster_bucket/ca.pem /ssl ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/ ssl registry.barklyprotects.com/ops/awscli s3 cp s3:// our_k8s_cluster_bucket/apiserver.pem /ssl ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/ ssl registry.barklyprotects.com/ops/awscli s3 cp s3:// our_k8s_cluster_bucket/apiserver-key.pem /ssl - name: kub_pods.service command: start content: | [Unit] Description=Writes kubernetes pod files to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target Before=kubelet.service [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c "/usr/bin/mkdir -p /etc/kubernetes/ssl" ExecStart=/usr/bin/docker run --net=host -v /etc/barkly/:/etc/ barkly/ -e K8S_VERSION='1.1.7' -e CLOUD_PROVIDER='--cloud-provider=aws' -e SERVICE_IP_RANGE="10.3.0.0/16" -e ADVERTISE_IP="$private_ipv4" -e ETCD_AUTOSCALE_GROUP_NAME="our_etcd_autoscaling_group_name" -v /srv/ kubernetes/manifests:/output_src -v /etc/kubernetes/manifests:/output_dst registry.barklyprotects.com/kubernetes/kub-master-pods
  • 23. Start Docker & Kubelet • Kubelet will wait for docker and flannel to be ready • Kubelet will load manifests for Master services outputted from previous container - name: docker.service command: start drop-ins: - name: 40-flannel.conf content: | [Unit] Requires=flanneld.service After=flanneld.service - name: kubelet.service command: start content: | [Unit] Requires=docker.service After=docker.service After=fluentd-elasticsearch.service [Service] ExecStartPre=/usr/bin/mkdir -p /var/log/containers ExecStart=/etc/bin/kubelet --hostname-override="$private_ipv4" --api_servers=http://127.0.0.1:8080 --register-node=false --allow-privileged=true --config=/etc/kubernetes/manifests --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --cloud-provider=aws --v=4 Restart=always RestartSec=10 [Install] WantedBy=multi-user.target
  • 24. Get Kubelet • /usr/bin & /usr/local/bin are read only in CoreOS - name: kubelet.service command: start drop-ins: - name: 10-download-binary.conf content: | [Service] ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet" write_files: # Since systemd needs these files before it will start - path: /etc/bin/download-k8s-binary permissions: '0755' content: | #!/usr/bin/env bash export K8S_VERSION=“v1.2.4" mkdir -p /etc/bin FILE=$1 if [ ! -f /usr/bin/$FILE ]; then curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly- kubernetes-builds/${K8S_VERSION}/bin/$FILE chmod +x /etc/bin/$FILE else # we check the version of the binary INSTALLED_VERSION=$(/etc/bin/$FILE --version) MATCH=$(echo "${INSTALLED_VERSION}" | grep -c "${K8S_VERSION}") if [ $MATCH -eq 0 ]; then # the version is different curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly- kubernetes-builds/${K8S_VERSION}/bin/$FILE chmod +x /etc/bin/$FILE fi fi
  • 25. Terraform to build • Similar to as before.. we reference our cloudinit script resource "aws_launch_configuration" "terraform_master" { name_prefix = "${var.environment}_master_conf-" image_id = "${var.coreos_ami}" instance_type = "t2.medium" key_name = "${var.key_name}" security_groups = ["$ {aws_security_group.terraform_master_sec_group.id}"] user_data = "${file("../cloud-config/output/master.yml")}" enable_monitoring = true ebs_optimized = false iam_instance_profile = "$ {aws_iam_instance_profile.terraform_master_role_profile.id}" root_block_device { volume_size = 20 } lifecycle { create_before_destroy = true } } resource "aws_autoscaling_group" "terraform_master" { name = "${var.environment}_master" launch_configuration = "$ {aws_launch_configuration.terraform_master.name}" availability_zones = ["us-east-1c"] max_size = "${var.capacities_master_max}" min_size = "${var.capacities_master_min}" health_check_grace_period = 300 desired_capacity = "${var.capacities_master_desired}" vpc_zone_identifier = ["${aws_subnet.master.id}"] force_delete = true load_balancers = ["${aws_elb.terraform_master.name}"] tag { key = "Name" value = "${var.environment}_master" propagate_at_launch = true } lifecycle { create_before_destroy = true } }
  • 27. Minion Node (now just Nodes) • Kubelet Service manages all other services. • Proxy • Pods
  • 28. Early Docker / Flannel • Same as master! #cloud-config coreos: units: - name: etcd.service command: stop - name: etcd2.service command: stop - name: early-docker.service command: start - name: kub_get_etcd.service command: start content: | [Unit] Description= Write K8s etcd urls to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c "/usr/bin/docker pull registry.barklyprotects.com/kubernetes/kub-get-etcd" ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/ barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{ KUB_ETCD_ASG }} > /etc/etcd_servers.env" - name: flanneld.service command: start drop-ins: - name: 10-environment_vars.conf content: | [Unit] After=kub_get_etcd.service [Service] ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE= $private_ipv4 >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/ flannel/options.env Restart=always RestartSec=10
  • 29. Kubelet • Similar to master - name: docker.service command: start drop-ins: - name: 40-flannel.conf content: | [Unit] Requires=flanneld.service After=flanneld.service - name: kubelet.service command: start content: | [Unit] Requires=docker.service After=docker.service After=fluentd-elasticsearch.service [Service] ExecStartPre=/usr/bin/mkdir -p /var/log/containers ExecStart=/etc/bin/kubelet --api_servers=https://ourk8smaster.barkly.com --hostname-override="$private_ipv4" --register-node=true --allow-privileged=true --config=/etc/kubernetes/manifests --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --kubeconfig=/etc/kubernetes/worker-kubeconfig-kubelet.yaml --tls-cert-file=/etc/kubernetes/ssl/worker.pem --tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem --cloud-provider=aws --v=4 Restart=always RestartSec=10 [Install] WantedBy=multi-user.target drop-ins: - name: 10-download-binary.conf content: | [Service] ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet"
  • 30. Manifests can be hard coded • Minion manifests are far less dynamic and can be hard coded as CloudInit files write_files: - path: "/etc/kubernetes/manifests/kube-proxy.yaml" content: | apiVersion: v1 kind: Pod metadata: name: kube-proxy namespace: kube-system spec: hostNetwork: true containers: - name: kube-proxy image: registry.barklyprotects.com/kubernetes/hyperkube:v1.1.8 command: - /hyperkube - proxy - --master=https://ourk8smaster.barkly.com - --kubeconfig=/etc/kubernetes/worker-kubeconfig-proxy.yaml - --proxy-mode=iptables - --v=4 securityContext: privileged: true volumeMounts: - mountPath: /etc/ssl/certs name: "ssl-certs" - mountPath: /etc/kubernetes/worker-kubeconfig-proxy.yaml name: "kubeconfig" readOnly: true - mountPath: /etc/kubernetes/ssl name: "etc-kube-ssl" readOnly: true volumes: - name: "ssl-certs" hostPath: path: "/usr/share/ca-certificates" - name: "kubeconfig" hostPath: path: "/etc/kubernetes/worker-kubeconfig-proxy.yaml" - name: "etc-kube-ssl" hostPath: path: "/etc/kubernetes/ssl"
  • 31. Manifests can be hard coded • Minion manifests are far less dynamic and can be hard coded as CloudInit files - path: "/etc/kubernetes/worker-kubeconfig-proxy.yaml" content: | apiVersion: v1 kind: Config clusters: - name: local cluster: certificate-authority: /etc/kubernetes/ssl/ca.pem users: - name: kubelet user: client-certificate: /etc/kubernetes/ssl/worker.pem client-key: /etc/kubernetes/ssl/worker-key.pem contexts: - context: cluster: local user: kubelet name: kubelet-context current-context: kubelet-context - path: "/etc/kubernetes/worker-kubeconfig-kubelet.yaml" content: | apiVersion: v1 kind: Config clusters: - name: local cluster: certificate-authority: /etc/kubernetes/ssl/ca.pem users: - name: kubelet user: client-certificate: /etc/kubernetes/ssl/worker.pem client-key: /etc/kubernetes/ssl/worker-key.pem contexts: - context: cluster: local user: kubelet name: kubelet-context current-context: kubelet-context
  • 33. K8s does not so well: • Still relatively young, • Not extremely google-able • Leaves managing manifests to user with little guidance • Docker only (rkt coming soon) • No DNS support (coming soon) • SaltStack by default
  • 34. K8s does really well: • Keeping applications running • Managing failover, liveness,readiness of applications • Mounting volumes, load balancers (cloud resources) • Internal DNS, Namespacing • Basic secret / config management • Easily Horizontally Scalable services • Rolling updates / rollback via Deployments • Support via Github / Slack chat • Easy debugging of actual app problems via kubectl logs and kubectl exec
  • 35. Demo!