Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Deployment with Kubernetes, Docker and GitLab CI

5,500 views

Published on

The talk is about how we implemented continuous deployment at the University of Leipzig.

Published in: Software

Continuous Deployment with Kubernetes, Docker and GitLab CI

  1. 1. Continuous Deployment with Kubernetes, Docker and GitLab CI @alexander_kiel
 Clojure Berlin 2016
  2. 2. Outline • Continuous Deployment why? • Docker • Kubernetes • Sample Clojure Service • Deploy with GitLabCI
  3. 3. Continuous Deployment • What do we want? • Increase responsiveness • Decrease time to market • Gain confidence by deploying often in small amounts • How to achieve that? • Automate everything • Always deploy the master into production • Use feature toggles when needed
  4. 4. Simple Git Workflow • Works for in-house apps • not for libs or shipping apps • No versions, no tags, just SHA’s • Latest commit on master is always deployed to production • Feature/fix branches are merged when ready master feature/fix branches 1ebb95d be61dda 6e4010d
  5. 5. Docker • Like VM’s but much more light-weight and shippable • Runs on Linux, executes processes in an isolated environment (resource limitation, filesystem, network) • Container principle: Can contain everything, but looks the same from the outside • A container platform can run every container • Developers have max. freedom what to do • In contrast: PaaS like Heroku - has to support the language
  6. 6. Kubernetes • Container runtime platform • Originally designed by Google - now Open Source • One of the most active projects on GitHub - 20,000 stars, 40,000 commits, 15,000 issues, 200 releases • Alternatives: Apache Mesos, Docker Swarm (lacks features)
  7. 7. Kubernetes Architecture k8s-master-1 k8s-master-2 k8s-master-3 load-balancer-1 load-balancer-2 DNS RR k8s-worker-1 proxy app-1 k8s-worker-2 proxy app-2 k8s-worker-n proxy app-k etcd cluster
 quorum HAProxy • Runs on VMware ESX • CoreOS Linux • Single YAML file as configuration • Everything in containers
  8. 8. Kubernetes - Pods • A Pod is a deployable unit in Kubernetes • Pods can contain multiple containers • Containers inside a Pod share on port space, can use localhost and can communicate via IPC and shared memory • Idea: one process per container - many cooperating processes in one Pod apiVersion: v1
 kind: Pod
 metadata:
 name: <pod-name>
 labels:
 <key>: <value>
 spec:
 containers:
 - name: <container-name>
 image: <container-image>
 ports:
 - containerPort: 80
 env:
 - name: <key>
 value: <value>
  9. 9. Kubernetes - Deployments • A Deployment ensures that certain number of Pods are always running • It consists of a Pod template and the number of replicas • It supports hot-redeployments by changing parts of the Pod template • Horizontal scaling is possible apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
 name: <deployment-name>
 spec:
 replicas: 2
 template:
 metadata: labels: <key>: <value> spec: containers: - name: <container-name> image: <container-image> ports: - containerPort: 80 env: - name: <key> value: <value>
  10. 10. Kubernetes - Services • Kubernetes uses an overlay network to provide different address spaces (we use flannel) • Every Pod has an IP address - but it changes every time one is created • Services provide a stable IP address for groups of Pods • Service names are resolvable by an internal DNS • Service selectors are used to match Pods according to there labels apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP selector: service: clojure- berlin-2016
  11. 11. Kubernetes - External Access • Kubernetes networks are internal only • External access through load balancers necessary • Certain Platforms like Google Compute Engine provide load balancer integration with Kubernetes • We have our own solution as a combination of HAProxy and Kubernetes NodePort • Kubernetes Services with type NodePort are exposed on every worker under a certain port frontend http bind 0.0.0.0:80 mode http option httplog acl host_clj hdr(host) clj.<domain> use_backend clj if host_clj backend clj mode http balance roundrobin option httplog server worker-1 <ip>:32599 check server worker-2 <ip>:32599 check
  12. 12. Deployment Lifecycle GitLab CI Source Code build test Kubernetes Test Cluster Kubernetes Prod Cluster automatic deployment manual deployment git push
  13. 13. Sample Clojure Service • .gitlab-ci.yml • Like .travis.yml contains instructions for GitLabCI how to test, build and deploy • Dockerfile • Instructions for Docker how to build the image of the app • Artifact of the build is a docker image - not uberjar • kube-deployment.yml • Kubernetes deployment instructions • kube-svc.yml • Kubernetes service description https://github.com/alexanderkiel/clojure-berlin-2016
  14. 14. The Core Namespace (ns clojure-berlin-2016.core (:require [aleph.http :as http] [clojure.core.async :refer [<!! chan]])) (defn -main [& args] (-> (fn [_] {:status 200 :body "Clojure Berlin 2016"}) (http/start-server {:port 8080})) (<!! (chan))) • A simple web server returning "Clojure Berlin 2016"
  15. 15. The Leiningen Project File (defproject clojure-berlin-2016 "<VERSION>" :dependencies [[aleph "0.4.1"] [org.clojure/clojure "1.8.0"] [org.clojure/core.async "0.2.395"]] :main clojure-berlin-2016.core) • <VERSION> is replaced at build time by the Git SHA • :main is for lein run to work
  16. 16. .gitlab-ci.yml - test/build image: clojure:lein-2.7.1 stages: - test - build - deploy test: stage: test tags: - docker script: - lein test build: stage: build tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" project.clj - docker build -t clojure-berlin-2016:$CI_BUILD_REF . - docker push clojure-berlin-2016:$CI_BUILD_REF
  17. 17. .gitlab-ci.yml - deploy branch deploy-branch: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml except: - master when: manual • Used to test a feature/fix branch in a full environment
  18. 18. .gitlab-ci.yml - deploy test deploy-master: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml only: - master
  19. 19. .gitlab-ci.yml - deploy prod deploy-prod: stage: deploy environment: prod image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-prod-a - kubectl apply -f kube-deployment.yml only: - master when: manual
  20. 20. Docker file FROM clojure:lein-2.7.1 COPY src /app/src COPY project.clj /app/ WORKDIR /app RUN lein with-profile production deps EXPOSE 80 CMD ["lein", "with-profile", "production", "run"] • Just copy the sources into the container • Use Leiningen itself to run in production
  21. 21. kube-deployment.yml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: clojure-berlin-2016 spec: replicas: 2 template: metadata: labels: app: lens service: clojure-berlin-2016 spec: containers: - name: clojure-berlin-2016 image: dreg.life.uni-leipzig.local/clojure-berlin-2016:<VERSION> ports: - containerPort: 8080 resources: requests: cpu: "125m" memory: "1Gi" limits: cpu: 1 memory: "2Gi"
  22. 22. kube-svc.yml apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 8080 protocol: TCP selector: service: clojure-berlin-2016
  23. 23. Steps to Follow • Create the Kubernetes Service • kubectl create -f kube-svc.yml • Edit HAProxy Config • add rules and backend for the service • Push to GitLab • git push
  24. 24. Pipeline in GitLab CI
  25. 25. Deployment in GitLabCI
  26. 26. Environments in GitLabCI • Very good visibility of wich commit is deployed in which environment right now • Manual deployment to prod possible
  27. 27. Environment History • Easy to see when what commit was deployed • Rollback possible
  28. 28. Numbers • Our team has 4 developers • We run 2 Kubernetes clusters (test and prod) with about 96 GB RAM and and 24 vCPU’s each • We run about 60 pods in production • We have other services like central log aggregation running using Fluentd and Elasticsearch/Kibana
  29. 29. Thank You • Sample Project on Github
 https://github.com/alexanderkiel/clojure-berlin-2016 • Twitter
 @alexander_kiel • Mail
 alexanderkiel@gmx.net

×