Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernetes at Spreadshirt - First steps to production

907 views

Published on

This presentation describes how we at Spreadshirt got started on our adventure into Docker-land which finally led to introducing Kubernetes for container orchestration.

Published in: Engineering
  • Be the first to comment

Kubernetes at Spreadshirt - First steps to production

  1. 1. Spreadshirt Kubernetes at Spreadshirt First steps to production Jens Hadlich, Chief Architect
  2. 2. Spreadshirt Spreadshirt WE PRINT IDEAS
  3. 3. Spreadshirt Not all that long ago … We want to build a service with Ruby … Why? … and the other team was allowed to build a service with Node.js … … Sure, I already forgot. Hmm … OK. Let‘s try. Ruby is the right tool for the job ... We could build it in Java but that would be too much … Other companies use it for years already … It‘s a very simple service … … … … 3
  4. 4. Spreadshirt Some time later … To run it, you just need some to install some gems. Didn‘t work. Sure not. You need Ruby in Version X. X? You can only have Y. What‘s a gem? … How about using Docker? … … Ecosystem man. … 4
  5. 5. Spreadshirt 5 Docker
  6. 6. Spreadshirt Docker is nice, but … * Number of Running Containers 1 5 10 50 100 * without container orchestration or additional tooling 6
  7. 7. Spreadshirt Docker is nice, but … * Number of Running Containers 1 5 10 50 100 * without container orchestration or additional tooling 7
  8. 8. Spreadshirt Docker is nice, but … * Number of Running Containers 1 5 10 50 100 * without container orchestration or additional tooling 8
  9. 9. Spreadshirt Docker is nice, but … * Number of Running Containers 1 5 10 50 100 * without container orchestration or additional tooling 9
  10. 10. Spreadshirt Docker is nice, but … * Number of Running Containers 1 5 10 50 100 * without container orchestration or additional tooling 10
  11. 11. Spreadshirt Docker is nice, but … * 11 Number of Running Containers 1 5 10 50 100 * without container orchestration or additional tooling
  12. 12. Spreadshirt 12 Kubernetes
  13. 13. Spreadshirt Kubernetes for Container Orchestration The promise ... • Easily bring up new components and services • Easily manage hundreds or thousands of containers • Easily add capacity • Better resource utilization • (Auto-)scale 13
  14. 14. Spreadshirt Preparation Think about … • Monitoring • Logging • Metrics • Configuration management • Build & Deployment pipelines • Your application stack 14
  15. 15. Spreadshirt Tool Ecosystem at Spreadshirt 15
  16. 16. Spreadshirt Tool Ecosystem at Spreadshirt (cont’d) 16 VxLAN (and even more …)
  17. 17. Spreadshirt 17
  18. 18. Spreadshirt Build git commit git push .gitlab-ci.yml * io.fabric8/docker-maven-plugin * 18
  19. 19. Spreadshirt Deploy Manifest files kubectl Service IP 19
  20. 20. Spreadshirt Kuh (Kubernetes hiera) • Homegrown preprocessor for k8s manifest files • Why? Avoid yet another big project (change configuration management) • Bridge between Puppet/hiera (our “old” world) and Kubernetes (the “new” world) – peaceful coexistence 20
  21. 21. Spreadshirt Puppet/hiera 21
  22. 22. Spreadshirt Kuh input structure ├── deployment │ ├── data │ │ ├── common.yaml │ │ ├── ops.yaml │ │ ├── qa.yaml │ │ ├── <team>.yaml │ │ └── ... │ ├── src │ │ ├── config │ │ │ └── application-config.yaml │ │ └── infra │ │ └── k8s.yaml hiera “data” files like with Puppet add to your git project 22 k8s manifest templates
  23. 23. Spreadshirt Kubernetes manifest templates with Kuh apiVersion: v1 kind: Service metadata: name: my-service labels: app: my-service spec: type: NodePort clusterIP: 192.168.X.Y ports: - port: <%= scope().call_function('hiera',['my_service::http_port']) %> name: http ... Retrieve the value from hiera* 23 * yeah, we know this looks ugly …
  24. 24. Spreadshirt Kuh output ├── deployment │ ├── data │ ├── src │ └── target │ ├── config │ │ └── application-config.yaml │ └── k8s │ ├── configmap │ │ └── application-config.yaml │ └── infra │ └── k8s.yaml 24 Final k8s manifest files
  25. 25. Spreadshirt Kubernetes Manifests • Deployment • ConfigMap • Secret • Service • (more to come) 25
  26. 26. Spreadshirt Docker registry & images • Nexus 3  Java / Maven repository  Own Docker registry - Nice WebUI - Traffic* • Docker images  latest tag, override  Other tags: - ${git.commit.time}.${git.commit.id} e.g. 2017-03-29.a5344e3501d591e4c7908d97a044d3d0f38ddb43  to be able to clean up old images and better readability 26 * pushing and pulling Docker images can generate significant network traffic
  27. 27. Spreadshirt Stateful Applications • Not at the moment! • Databases, Elasticsearch, … are “external” • Avoid local storage • Use Web-APIs (e.g. S3) 27
  28. 28. Spreadshirt Open issues We are still in learning mode! • Improve high availability for Kubernetes  2 separate Kubernetes clusters - Single master: if it fails – restart - Clustered etcd  Puppetized => spin up a new cluster within minutes • How to manage storage / stateful services?  E.g. DB, Elasticsearch 28
  29. 29. Spreadshirt Open issues (cont’d) • Security patches / lifecycle of Docker images / containers within k8s  What’s running inside my container?  Manage re-build / re-deploy after a certain time?  Trust 3rd party base images? • Reduce complexity • Revisit custom HAProxy vs. built-in Ingress & Co. 29
  30. 30. Spreadshirt Nice tools • sysdig / csysdig 30
  31. 31. Spreadshirt Thank You 31 jns@spreadshirt.com
  32. 32. Spreadshirt Links • https://sysdig.com/blog/digging-into-kubernetes-with-sysdig/

×