Downloaded 10 times
![32
STACKSET CRD
apiVersion: zalando.org/v1
kind: StackSet
...
spec:
ingress:
hosts: ["foo.example.org"]
backendPort: 8080
stackLifecycle:
scaledownTTLSeconds: 1800
limit: 5
stackTemplate:
spec:
podTemplate:
...
github.com/zalando-incubator/stackset-controller](https://image.slidesharecdn.com/2019-04-18developerexperienceatzalando-cncfendusersig-dx-190418193313/85/Developer-Experience-at-Zalando-CNCF-End-User-SIG-DX-32-320.jpg)
Uploaded byHenning Jacobs
Developer Experience at Zalando - CNCF End User SIG-DX
This document summarizes Zalando's approach to developer experience and Kubernetes operations. Key aspects include: - Developers build and deploy applications using Kubernetes as the primary interface. They are responsible for operations through an "on-call" model. - Tools and platforms are provided to automate common tasks like builds, deployments, monitoring and scaling. These aim to improve developer productivity while ensuring correctness, security and cost efficiency. - Open source projects developed by Zalando help provision and manage Kubernetes clusters and applications at large scale, with over 1000 developers and 200 teams using Kubernetes internally.
More Related Content
Similar to Developer Experience at Zalando - CNCF End User SIG-DX
Developer Experience at Zalando - CNCF End User SIG-DX
- 1. CNCF END USER SIG-DX 2019-04-18 HENNINGJACOBS @try_except_ Developer Experience at Zalando
- 2. 2 EUROPE’S LEADING ONLINEFASHION PLATFORM
- 3. 3 ZALANDO AT AGLANCE ~ 5.4billion EUR revenue 2018 > 250 million visits per month > 15.000 employees in Europe > 79% of visits via mobile devices > 26 million active customers > 300.000 product choices ~ 2.000 brands 17 countries
- 4.
- 5. 5 YOU BUILD IT,YOU RUN IT The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. - A Conversation with Werner Vogels, ACM Queue, 2006
- 6. 6 ON-CALL: YOU OWNIT, YOU RUN IT When things are broken, we want people with the best context trying to fix things. - Blake Scrivener, Netflix SRE Manager
- 7. 7 KUBERNETES @ ZALANDO 114 clusters 1400~ nodes Since Oct2016 Node Autoscaling From v1.4 to v1.12 Default Deployment Target
- 8.
- 9. 9 DEVELOPER JOURNEY Consistent story thatmodels all aspects of SW dev
- 10.
- 11. 11 Developer Journey Correctness Compliance GDPR Security Cost Efficiency 24x7 OnCall Governance Resilience Capacity ...
- 12. 12 DEVELOPER PRODUCTIVITY Code BuildTest Deploy OperateSetup Cloud Native Application Runtime
- 14.
- 15.
- 17.
- 20.
- 21.
- 23.
- 24.
- 25. 25 DEPLOYMENT CONFIGURATION ├── deploy/apply │├── deployment.yaml │ ├── credentials.yaml # Zalando IAM │ ├── ingress.yaml │ └── service.yaml └── delivery.yaml # Zalando CI/CD
- 26. 26 INGRESS.YAML kind: Ingress metadata: name: "..." spec: rules: #DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80
- 27. 27 TEMPLATING: MUSTACHE kind: Ingress metadata: name:"..." spec: rules: # DNS name your application should be exposed on - host: "{{{APPLICATION}}}.example.org" http: paths: - backend: serviceName: "{{{APPLICATION}}}" servicePort: 80
- 28.
- 29.
- 30.
- 31.
- 32. 32 STACKSET CRD apiVersion: zalando.org/v1 kind:StackSet ... spec: ingress: hosts: ["foo.example.org"] backendPort: 8080 stackLifecycle: scaledownTTLSeconds: 1800 limit: 5 stackTemplate: spec: podTemplate: ... github.com/zalando-incubator/stackset-controller
- 33. 33 TRAFFIC SWITCHING STEPSIN CDP github.com/zalando-incubator/stackset-controller
- 34. 34 EMERGENCY ACCESS SERVICE Getemergency access by referencing existing Incident ticket: zkubectl cluster-access request --emergency -i INC REASON Get privileged production access via 4-eyes: zkubectl cluster-access request REASON zkubectl cluster-access approve USERNAME
- 35.
- 36. 36 CLOUD FORMATION VIACI/CD ├── deploy/apply │ ├── deployment.yaml # Kubernetes │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml │ ├── kube-secret.yaml │ └── kube-service.yaml └── delivery.yaml # CI/CD config "Infrastructure as Code"
- 37. 37 ZALANDO IAM/OAUTH VIACRD apiVersion: zalando.org/v1 kind: PlatformCredentialsSet .. spec: application: my-app tokens: read-only: privileges: - com.zalando::foobar.read clients: employee: grant: authorization-code realm: users redirectUri: https://example.org/auth/callback
- 38. 38 POSTGRES OPERATOR Application tomanage PostgreSQL clusters on Kubernetes >700 clusters running on Kubernetes github.com/zalando/postgres-operator
- 39. Elasticsearch in Kubernetes Elasticsearch 2.500vCPUs 1 TB RAM github.com/zalando-incubator/es-operator/
- 40. 40 SUMMARY • Application Bootstrapping •Git as source of truth and UI • 4-eyes principle for master/production • Extensible Kubernetes API as primary interface • OAuth/IAM credentials • PostgreSQL • CloudFormation for proprietary AWS services
- 41. 41 DELIVERY PERFORMANCE METRICS •Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate https://srcco.de/posts/accelerate-software-delivery-performance.html
- 42. 42 CONTAINERS From "Accelerate: TheScience of Lean Software and DevOps"
- 43. 43 DELIVERY PERFORMANCE METRICS •Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate ≙ Commit to Prod ≙ Deploys/week/dev ≙ MTRS from incidents ≙ n/a
- 44. “.. means establishingempathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience.” - ThoughtWorks Technology Radar
- 46.
- 47. 47 DOCUMENTATION "Documentation is hardto find" "Documentation is not comprehensive enough" "Remove unnecessary complexity and obstacles." "Get the documentation up to date and prepare use cases" "More and more clear documentation" "More detailed docs, example repos with more complicated deployments."
- 48. 48 DOCUMENTATION • Restructure following https://www.divio.com/en/blog/documentation/ •Concepts • How Tos • Tutorials • Reference • Global Search • Weekly Health Check: Support → Documentation
- 50. 50 NEWSLETTER "You can now.." •You can now benefit from the most recent Kubernetes 1.12 features, e.g. .. • You can now analyse your Kotlin project with SonarQube and upload your Scala code coverage report to SonarQube
- 51.
- 52. 52 TESTIMONIALS “So, thank you,Team Automata, for listening to our community, taking our upvotes in consideration when developing new solutions and building every day 'the first CI that doesn't suck'.” - a user, October 2018
- 53.
- 54.
- 55.
- 56.
- 57. 57 RESOURCE REPORT: TEAMS Sortingteams by Slack Costs github.com/hjacobs/kube-resource-report
- 58.
- 59.
- 60.
- 61. 61 ZALANDO: DECISION 1. ForbidMemory Overcommit • Implement mutating admission webhook • Set requests = limits 2. Disable CPU CFS Quota in all clusters • --cpu-cfs-quota=false
- 62.
- 63. 63 CLUSTER PROVISIONING CLUSTER LIFECYCLEMANAGER (CLM) ADMIN create apply manifests provision resources create CF stack CLUSTER REGISTRY CLM API ... ... ... CloudFormation API github.com/zalando-incubator/cluster-lifecycle-manager github.com/zalando-incubator/kubernetes-on-aws
- 64.
- 65. 65 VPA FOR PROMETHEUS apiVersion:poc.autoscaling.k8s.io/v1alpha1 kind: VerticalPodAutoscaler metadata: name: prometheus-vpa namespace: kube-system spec: selector: matchLabels: application: prometheus updatePolicy: updateMode: Auto CPU/memory
- 66. 66 VERTICAL POD AUTOSCALER limit/requestsadapted by VPA
- 67. 67 HORIZONTAL POD AUTOSCALING(CUSTOM METRICS) Queue Length Prometheus Query Ingress Req/s ZMON Check github.com/zalando-incubator/kube-metrics-adapter
- 68.
- 69. 69 DOWNSCALING DURING OFF-HOURS DEFAULT_UPTIME="Mon-Fri07:30-20:30 CET" annotations: downscaler/exclude: "true" github.com/hjacobs/kube-downscaler
- 70. 70 KUBERNETES JANITOR ● TTLand expiry date annotations, e.g. ○ set time-to-live for your test deployment ● Custom rules, e.g. ○ delete everything without "app" label after 7 days github.com/hjacobs/kube-janitor
- 71. 71 JANITOR TTL ANNOTATION #let's try out nginx, but only for 1 hour kubectl run nginx --image=nginx kubectl annotate deploy nginx janitor/ttl=1h github.com/hjacobs/kube-janitor
- 72. 72 CUSTOM JANITOR RULES #require "app" label for new pods starting April 2019 - id: require-app-label-april-2019 resources: - deployments - statefulsets jmespath: "!(spec.template.metadata.labels.app) && metadata.creationTimestamp > '2019-04-01'" ttl: 7d github.com/hjacobs/kube-janitor
- 73. 73 EC2 SPOT NODES 72%savings
- 74. 74 SPOT ASG /LAUNCH TEMPLATE Not upstream in cluster-autoscaler (yet)
- 75. 75 OPEN SOURCE Kubernetes onAWS github.com/zalando-incubator/kubernetes-on-aws AWS ALB Ingress controller github.com/zalando-incubator/kube-ingress-aws-controller External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando/postgres-operator Kubernetes Resource Report github.com/hjacobs/kube-resource-report Kubernetes Downscaler github.com/hjacobs/kube-downscaler Kubernetes Janitor github.com/hjacobs/kube-janitor
- 76. 76 MORE INFO ● DevOpsGathering 2019: Ensuring Kubernetes Cost Efficiency across (many) Clusters (slides) ● DevOpsCon Munich 2018: Running Kubernetes in Production: A Million Ways to Crash Your Cluster ● HighLoad++ Moscow 2018: Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency (slides) ● DevOps Lisbon Meetup 2018: Kubernetes at Zalando kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/public-presentations.html
- 77. QUESTIONS? HENNING JACOBS HEAD OF DEVELOPERPRODUCTIVITY henning@zalando.de @try_except_ Illustrations by @01k