Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Container Days Boston - Kubernetes in production

1,356 views

Published on

We will dive into our year+ long journey into K8s

Published in: Software
  • Be the first to comment

Container Days Boston - Kubernetes in production

  1. 1. Kubernetes in Production, From the Ground Up Mike Splain | @mikesplain
  2. 2. About Me • Lead DevOps Engineer at Barkly • Formerly PayPal / eBay & OurStage • Running applications at scale with the least amount of human interaction • Current Technologies: Kubernetes, Docker, Terraform, Ansible • Previous Technologies: Chef, Test-Kitchen….
  3. 3. Resources • docs.kubernetes.io • coreos.com/kubernetes/docs • bostonkubernetes.com • slack.kubernetes.io • github.com/mikesplain/etcd_demo
  4. 4. Problem:
  5. 5. Our requirements for running our apps • Fast recovery without human intervention • Nodes are ephemeral • Autoscaling • Testable on developer machines • Infrastructure as an artifact
  6. 6. Technology choices for running K8s
  7. 7. Hasn’t someone else already built this? … March 2015
  8. 8. K8s official scripts • Simple bash scripts • “Just Works!” • Sets up Autoscaling group for minions • Uses Salt Problems? (at the time) • No etcd HA • No Master HA • Salt Master is coupled with K8s Master • Ubuntu / Fedora
  9. 9. CoreOS’s official scripts • Cool go app to start it! • Or Cloud formation? • But what now? Problems? (at the time) • No etcd HA • No Master HA • Lots of magic
  10. 10. “It can’t be that hard right?”
  11. 11. etcd
  12. 12. Note to self… you’re going too slow.
  13. 13. etcd • Easy. • “Just works” • Cluster discovery: • Discovery Service • DNS • ? #cloud-config coreos: etcd2: advertise-client-urls: "http://$public_ipv4:2379" initial-advertise-peer-urls: "http://$private_ipv4:2380" listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001" listen-peer-urls: "http://$private_ipv4:2380,http://$private_ipv4:7001" discovery-token: “<token here>” units: - name: etcd2.service command: start update: reboot-strategy: none
  14. 14. etcd • etcd-aws-cluster
 https://github.com/MonsantoCo/ etcd-aws-cluster • Uses Autoscaling groups for discovery • Requires IAM Instance Roles #cloud-config coreos: etcd2: advertise-client-urls: "http://$public_ipv4:2379" initial-advertise-peer-urls: "http://$private_ipv4:2380" listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001" listen-peer-urls: "http://$private_ipv4:2380,http://$private_ipv4:7001" units: - name: etcd2.service command: stop - name: etcd-peers.service command: start content: | [Unit] Description=Write a file with the etcd peers that we should bootstrap to Requires=docker.service After=docker.service [Service] Restart=on-failure RestartSec=10 TimeoutStartSec=300 ExecStartPre=/usr/bin/docker pull registry.barklyprotects.com/ kubernetes/etcd-aws-cluster:latest ExecStartPre=/usr/bin/docker run --rm=true -v /etc/sysconfig/:/etc/ sysconfig/ registry.barklyprotects.com/kubernetes/etcd-aws-cluster:latest ExecStart=/usr/bin/systemctl start etcd2 write_files: - path: /etc/systemd/system/etcd2.service.d/30-etcd_peers.conf permissions: 0644 content: | [Service] # Load the other hosts in the etcd leader autoscaling group from file EnvironmentFile=/etc/sysconfig/etcd-peers
  15. 15. Terraform to launch etcd • References static cloud-init files resource "aws_launch_configuration" "terraform_etcd" { name_prefix = "${var.environment}_etcd_conf-" image_id = "${var.coreos_ami}" instance_type = "t2.small" key_name = "${var.key_name}" security_groups = ["$ {aws_security_group.terraform_etcd2_sec_group.id}"] user_data = "${file("../cloud-config/output/etcd.yml")}" enable_monitoring = true ebs_optimized = false iam_instance_profile = "$ {aws_iam_instance_profile.terraform_etcd_role_profile.id}" root_block_device { volume_size = 20 } lifecycle { create_before_destroy = true } } resource "aws_autoscaling_group" "terraform_etcd" { name = "${var.environment}_etcd" launch_configuration = "$ {aws_launch_configuration.terraform_etcd.name}" availability_zones = ["us-east-1c"] max_size = "${var.capacities_etcd_max}" min_size = "${var.capacities_etcd_min}" health_check_grace_period = 300 desired_capacity = "${var.capacities_etcd_desired}" vpc_zone_identifier = ["${aws_subnet.etcd.id}"] force_delete = true tag { key = "Name" value = "${var.environment}_etcd" propagate_at_launch = true } lifecycle { create_before_destroy = true } }
  16. 16. Masters
  17. 17. Master Node • Kubelet service • API Server • Controller Manager • Scheduler • Proxy • Flannel
  18. 18. Consider Master pods as additional artifact • Docker container that takes env variables • Outputs templated pods to disk for Kubelet to load • We use j2cli to template these files with little overhead apiVersion: v1 kind: Pod metadata: name: kube-podmaster namespace: kube-system spec: hostNetwork: true containers: - name: scheduler-elector image: gcr.io/google_containers/podmaster:1.1 imagePullPolicy: Always command: - /podmaster - --etcd-servers={{ ETCD_ENDPOINTS }} - --key=scheduler - --whoami={{ ADVERTISE_IP }} - --source-file=/src/manifests/kube-scheduler.yaml - --dest-file=/dst/manifests/kube-scheduler.yaml volumeMounts: - mountPath: /src/manifests name: manifest-src readOnly: true - mountPath: /dst/manifests name: manifest-dst
  19. 19. Master Cloud Config • Setup box for services needed for real Docker to run • Get etcd server IPs and write to file • Start flannel with those IPs #cloud-config coreos: units: - name: etcd.service command: stop - name: etcd2.service command: stop - name: early-docker.service command: start - name: kub_get_etcd.service command: start content: | [Unit] Description= Write K8s etcd urls to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c "/usr/bin/docker pull registry.barklyprotects.com/kubernetes/kub-get-etcd" ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/ barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{ KUB_ETCD_ASG }} > /etc/etcd_servers.env" - name: flanneld.service command: start drop-ins: - name: 10-environment_vars.conf content: | [Unit] After=kub_get_etcd.service [Service] ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE= $private_ipv4 >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/ flannel/options.env Restart=always RestartSec=10
  20. 20. Other config • Grab certs from S3 • Terraform only allows permissions to specific files • Format Master pod files to disk - name: kub_certs.service command: start content: | [Unit] Description=Writes kubernetes cluster certs to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target Before=kubelet.service [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c /usr/bin/mkdir -p /etc/kubernetes/ssl ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/ ssl registry.barklyprotects.com/ops/awscli s3 cp s3:// our_k8s_cluster_bucket/ca.pem /ssl ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/ ssl registry.barklyprotects.com/ops/awscli s3 cp s3:// our_k8s_cluster_bucket/apiserver.pem /ssl ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/ ssl registry.barklyprotects.com/ops/awscli s3 cp s3:// our_k8s_cluster_bucket/apiserver-key.pem /ssl - name: kub_pods.service command: start content: | [Unit] Description=Writes kubernetes pod files to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target Before=kubelet.service [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c "/usr/bin/mkdir -p /etc/kubernetes/ssl" ExecStart=/usr/bin/docker run --net=host -v /etc/barkly/:/etc/ barkly/ -e K8S_VERSION='1.1.7' -e CLOUD_PROVIDER='--cloud-provider=aws' -e SERVICE_IP_RANGE="10.3.0.0/16" -e ADVERTISE_IP="$private_ipv4" -e ETCD_AUTOSCALE_GROUP_NAME="our_etcd_autoscaling_group_name" -v /srv/ kubernetes/manifests:/output_src -v /etc/kubernetes/manifests:/output_dst registry.barklyprotects.com/kubernetes/kub-master-pods
  21. 21. Start Docker & Kubelet • Kubelet will wait for docker and flannel to be ready • Kubelet will load manifests for Master services outputted from previous container - name: docker.service command: start drop-ins: - name: 40-flannel.conf content: | [Unit] Requires=flanneld.service After=flanneld.service - name: kubelet.service command: start content: | [Unit] Requires=docker.service After=docker.service After=fluentd-elasticsearch.service [Service] ExecStartPre=/usr/bin/mkdir -p /var/log/containers ExecStart=/etc/bin/kubelet --hostname-override="$private_ipv4" --api_servers=http://127.0.0.1:8080 --register-node=false --allow-privileged=true --config=/etc/kubernetes/manifests --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --cloud-provider=aws --v=4 Restart=always RestartSec=10 [Install] WantedBy=multi-user.target
  22. 22. Get Kubelet • /usr/bin & /usr/local/bin are read only in CoreOS - name: kubelet.service command: start drop-ins: - name: 10-download-binary.conf content: | [Service] ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet" write_files: # Since systemd needs these files before it will start - path: /etc/bin/download-k8s-binary permissions: '0755' content: | #!/usr/bin/env bash export K8S_VERSION=“v1.2.4" mkdir -p /etc/bin FILE=$1 if [ ! -f /usr/bin/$FILE ]; then curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly- kubernetes-builds/${K8S_VERSION}/bin/$FILE chmod +x /etc/bin/$FILE else # we check the version of the binary INSTALLED_VERSION=$(/etc/bin/$FILE --version) MATCH=$(echo "${INSTALLED_VERSION}" | grep -c "${K8S_VERSION}") if [ $MATCH -eq 0 ]; then # the version is different curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly- kubernetes-builds/${K8S_VERSION}/bin/$FILE chmod +x /etc/bin/$FILE fi fi
  23. 23. Terraform to build • Similar to as before.. we reference our cloudinit script resource "aws_launch_configuration" "terraform_master" { name_prefix = "${var.environment}_master_conf-" image_id = "${var.coreos_ami}" instance_type = "t2.medium" key_name = "${var.key_name}" security_groups = ["$ {aws_security_group.terraform_master_sec_group.id}"] user_data = "${file("../cloud-config/output/master.yml")}" enable_monitoring = true ebs_optimized = false iam_instance_profile = "$ {aws_iam_instance_profile.terraform_master_role_profile.id}" root_block_device { volume_size = 20 } lifecycle { create_before_destroy = true } } resource "aws_autoscaling_group" "terraform_master" { name = "${var.environment}_master" launch_configuration = "$ {aws_launch_configuration.terraform_master.name}" availability_zones = ["us-east-1c"] max_size = "${var.capacities_master_max}" min_size = "${var.capacities_master_min}" health_check_grace_period = 300 desired_capacity = "${var.capacities_master_desired}" vpc_zone_identifier = ["${aws_subnet.master.id}"] force_delete = true load_balancers = ["${aws_elb.terraform_master.name}"] tag { key = "Name" value = "${var.environment}_master" propagate_at_launch = true } lifecycle { create_before_destroy = true } }
  24. 24. Minions
  25. 25. Minion Node (now just Nodes) • Kubelet Service manages all other services. • Proxy • Pods
  26. 26. Early Docker / Flannel • Same as master! #cloud-config coreos: units: - name: etcd.service command: stop - name: etcd2.service command: stop - name: early-docker.service command: start - name: kub_get_etcd.service command: start content: | [Unit] Description= Write K8s etcd urls to disk. Requires=early-docker.service After=early-docker.service Before=early-docker.target [Service] Type=oneshot Environment="DOCKER_HOST=unix:///var/run/early-docker.sock" ExecStart=/usr/bin/sh -c "/usr/bin/docker pull registry.barklyprotects.com/kubernetes/kub-get-etcd" ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/ barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{ KUB_ETCD_ASG }} > /etc/etcd_servers.env" - name: flanneld.service command: start drop-ins: - name: 10-environment_vars.conf content: | [Unit] After=kub_get_etcd.service [Service] ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE= $private_ipv4 >> /etc/flannel_etcd_servers.env" ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/ flannel/options.env Restart=always RestartSec=10
  27. 27. Kubelet • Similar to master - name: docker.service command: start drop-ins: - name: 40-flannel.conf content: | [Unit] Requires=flanneld.service After=flanneld.service - name: kubelet.service command: start content: | [Unit] Requires=docker.service After=docker.service After=fluentd-elasticsearch.service [Service] ExecStartPre=/usr/bin/mkdir -p /var/log/containers ExecStart=/etc/bin/kubelet --api_servers=https://ourk8smaster.barkly.com --hostname-override="$private_ipv4" --register-node=true --allow-privileged=true --config=/etc/kubernetes/manifests --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --kubeconfig=/etc/kubernetes/worker-kubeconfig-kubelet.yaml --tls-cert-file=/etc/kubernetes/ssl/worker.pem --tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem --cloud-provider=aws --v=4 Restart=always RestartSec=10 [Install] WantedBy=multi-user.target drop-ins: - name: 10-download-binary.conf content: | [Service] ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet"
  28. 28. Manifests can be hard coded • Minion manifests are far less dynamic and can be hard coded as CloudInit files write_files: - path: "/etc/kubernetes/manifests/kube-proxy.yaml" content: | apiVersion: v1 kind: Pod metadata: name: kube-proxy namespace: kube-system spec: hostNetwork: true containers: - name: kube-proxy image: registry.barklyprotects.com/kubernetes/hyperkube:v1.1.8 command: - /hyperkube - proxy - --master=https://ourk8smaster.barkly.com - --kubeconfig=/etc/kubernetes/worker-kubeconfig-proxy.yaml - --proxy-mode=iptables - --v=4 securityContext: privileged: true volumeMounts: - mountPath: /etc/ssl/certs name: "ssl-certs" - mountPath: /etc/kubernetes/worker-kubeconfig-proxy.yaml name: "kubeconfig" readOnly: true - mountPath: /etc/kubernetes/ssl name: "etc-kube-ssl" readOnly: true volumes: - name: "ssl-certs" hostPath: path: "/usr/share/ca-certificates" - name: "kubeconfig" hostPath: path: "/etc/kubernetes/worker-kubeconfig-proxy.yaml" - name: "etc-kube-ssl" hostPath: path: "/etc/kubernetes/ssl"
  29. 29. Manifests can be hard coded • Minion manifests are far less dynamic and can be hard coded as CloudInit files - path: "/etc/kubernetes/worker-kubeconfig-proxy.yaml" content: | apiVersion: v1 kind: Config clusters: - name: local cluster: certificate-authority: /etc/kubernetes/ssl/ca.pem users: - name: kubelet user: client-certificate: /etc/kubernetes/ssl/worker.pem client-key: /etc/kubernetes/ssl/worker-key.pem contexts: - context: cluster: local user: kubelet name: kubelet-context current-context: kubelet-context - path: "/etc/kubernetes/worker-kubeconfig-kubelet.yaml" content: | apiVersion: v1 kind: Config clusters: - name: local cluster: certificate-authority: /etc/kubernetes/ssl/ca.pem users: - name: kubelet user: client-certificate: /etc/kubernetes/ssl/worker.pem client-key: /etc/kubernetes/ssl/worker-key.pem contexts: - context: cluster: local user: kubelet name: kubelet-context current-context: kubelet-context
  30. 30. Results
  31. 31. K8s does not so well: • Still relatively young, • Not extremely google-able • Leaves managing manifests to user with little guidance • Docker only (rkt coming soon) • No DNS support (coming soon) • SaltStack by default
  32. 32. K8s does really well: • Keeping applications running • Managing failover, liveness,readiness of applications • Mounting volumes, load balancers (cloud resources) • Internal DNS, Namespacing • Basic secret / config management • Easily Horizontally Scalable services • Rolling updates / rollback via Deployments • Support via Github / Slack chat • Easy debugging of actual app problems via kubectl logs and kubectl exec
  33. 33. Demo!
  34. 34. https://github.com/mikesplain/etcd_demo Questions?

×