Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Why we don’t use the Term DevOps: the Journey to a Product Mindset - Destination AARhus

194 views

Published on

While the adoption of DevOps makes teams move faster with reduced dependency on central operations, it can constrain teams who lack the skills to self-manage the full application and infrastructure stack.



The way to overcome this challenge is creating an internal platform and treating it as a world-class product offering. “Applying product management to internal platforms means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience”, via ThoughtWorks Technology Radar.



In this talk, Henning Jacobs will walk you through how Zalando adopted a customer-first mindset with regards to its developer tooling. He will show the effect on developer satisfaction when internal platforms are given the same respect as external product offerings. Henning will furthermore tell his story about how Zalando moved from a classical infrastructure team to a product mindset with strong focus on building a world-class developer experience. Henning shares both their learnings and challenges going through this transition, and the impact it has on the daily life of Zalando’s customers (developers).

This talk was given in Aarhus on 4th of June 2019.

Published in: Technology
  • Be the first to comment

Why we don’t use the Term DevOps: the Journey to a Product Mindset - Destination AARhus

  1. 1. Why we don't use the term DevOps DESTINATION AARHUS 2019-06-04 HENNING JACOBS @try_except_
  2. 2. 2 EUROPE’S LEADING ONLINE FASHION PLATFORM
  3. 3. 3 ZALANDO AT A GLANCE ~ 5.4billion EUR revenue 2018 > 250 million visits per month > 15.000 employees in Europe > 79% of visits via mobile devices > 26 million active customers > 300.000 product choices ~ 2.000 brands 17 countries
  4. 4. 4 THE BIRTH OF DEVOPS - 2009 “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” DevOps Days Belgium https://devops.com/the-origins-of-devops-whats-in-a-name/
  5. 5. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
  6. 6. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
  7. 7. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
  8. 8. 8 THE PHOENIX PROJECT - 2013 "The Three Ways" 1. Systems Thinking 2. Amplify Feedback Loops 3. Culture of Continual Experimentation And Learning
  9. 9. 9 WHAT HAPPENED NEXT
  10. 10. 10 CARGO CULTING
  11. 11. 11 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  12. 12. 12 DevOps is normal
  13. 13. 13
  14. 14. 14 WHAT HAPPENED NEXT
  15. 15. 15 WHAT HAPPENED NEXT
  16. 16. 16 DEV + OPS ⇒ DEVOPS Devs "DevOps" Team Ops
  17. 17. 17 DEV + OPS ⇒ ENGINEERING Devs "DevOps" Team Ops Engineering Team
  18. 18. 18 PRODUCT DEVELOPMENT TEAMS "Great products emerge from empowered, fully cross-functional teams." - John Cutler, 2019 https://amplitude.com/blog/on-being-product-led
  19. 19. 19 DEVOPS IS It is a culture, in which people work together to improve the product delivery cycle.
  20. 20. 20 DEVOPS ANTI-PATTERNS Dev and Ops Silos DevOps Team Silo Dev Don't Need Ops DevOps as Tools Team Rebranded SysAdmin Ops Embedded in Dev Team web.devopstopologies.com
  21. 21. 21
  22. 22. 22 TEAM INTERACTION MODES • Collaboration: 2 teams working together • X-as-a-Service: 1 provides, 1 consumes • Facilitating: 1 team helps another Product teams need a family too - DevOpsDays Portugal
  23. 23. 23 A BRIEF HISTORY OF ZALANDO TECH
  24. 24. 24 2010 "Sysop-Test" "QA-Test"
  25. 25. 25 DEVOPS IS ABOUT SKIN IN THE GAME Absence of skin in the game: People who are isolated from the impacts of their decisions do not learn. They remain captive to their erroneous ideas about how the world works.
  26. 26. 26 2013: SELF SERVICE
  27. 27. 27 2015: RADICAL AGILITY AWS STUPS DOCKER DEPLOY SSH ACCESS AUDIT REPORTS FULL AWS ACCESS Teams have admin access & full responsibility
  28. 28. 28 2015: ISOLATED AWS ACCOUNTS Internet *.abc.example.org *.xyz.example.org Team ABC Team XYZ EC2EC2 ELBELB EC2
  29. 29. 29 Platform > 1100 developers > 200 development teams 2019
  30. 30. 30 YOU BUILD IT, YOU RUN IT The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. - A Conversation with Werner Vogels, ACM Queue, 2006
  31. 31. 31 ON-CALL: YOU OWN IT, YOU RUN IT When things are broken, we want people with the best context trying to fix things. - Blake Scrivener, Netflix SRE Manager
  32. 32. 32 "SKIN IN THE GAME"
  33. 33. 33 DEVELOPER JOURNEY Consistent story that models all aspects of SW dev
  34. 34. 34 Developer Journey
  35. 35. 35 Developer Journey Correctness Compliance GDPR Security Cost Efficiency 24x7 On Call Governance Resilience Capacity ...
  36. 36. 36 DEVELOPER PRODUCTIVITY Code Build Test Deploy OperateSetup Cloud Native Application Runtime
  37. 37. 37 CLOUD NATIVE .. uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. Cloud native technologies enable software developers to build great products faster. - https://www.cncf.io/
  38. 38. 38 CONTAINERS END-TO-END Code Build Test Deploy OperateSetup Cloud Native Application Runtime
  39. 39. 39 CONTAINERS
  40. 40. 40 CONTAINERS
  41. 41. 41 PLATFORM AS A PRODUCT "A good platform is treated as a product (reliable, usable, fit for purpose)." - Manuel Pais, 2019 Product teams need a family too - DevOpsDays Portugal
  42. 42. 43 PLAN & SETUP
  43. 43. 44 Plan Stories Rules of Play Tech Radar
  44. 44. 46 Setup Application Bootstrapping
  45. 45. 49 BUILD & TEST
  46. 46. 50 CDPGit code push CONTINUOUS DELIVERY PLATFORM: BUILD
  47. 47. 52 DEPLOY
  48. 48. 53 Deploy Kubernetes
  49. 49. 54 DEPLOYMENT CONFIGURATION ├── deploy/apply │ ├── deployment.yaml │ ├── credentials.yaml # Zalando IAM │ ├── ingress.yaml │ └── service.yaml └── delivery.yaml # Zalando CI/CD
  50. 50. 55 INGRESS.YAML kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80
  51. 51. 56 TEMPLATING: MUSTACHE kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "{{{APPLICATION}}}.example.org" http: paths: - backend: serviceName: "{{{APPLICATION}}}" servicePort: 80
  52. 52. 57 CONTINUOUS DELIVERY PLATFORM
  53. 53. 58 CDP: DEPLOY "glorified kubectl apply"
  54. 54. 59 CDP: OPTIONAL APPROVAL
  55. 55. 60 STACKSET: TRAFFIC SWITCHING github.com/zalando-incubator/stackset-controller
  56. 56. 61 STACKSET CRD kind: StackSet ... spec: ingress: hosts: ["foo.example.org"] backendPort: 8080 stackLifecycle: scaledownTTLSeconds: 1800 limit: 5 stackTemplate: spec: podTemplate: ... github.com/zalando-incubator/stackset-controller
  57. 57. 62 TRAFFIC SWITCHING STEPS IN CDP github.com/zalando-incubator/stackset-controller
  58. 58. 63 EMERGENCY ACCESS SERVICE Emergency access by referencing Incident zkubectl cluster-access request --emergency -i INC REASON Privileged production access via 4-eyes zkubectl cluster-access request REASON zkubectl cluster-access approve USERNAME
  59. 59. 64 INTEGRATIONS
  60. 60. 65 CLOUD FORMATION VIA CI/CD ├── deploy/apply │ ├── deployment.yaml # Kubernetes │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml │ ├── kube-secret.yaml │ └── kube-service.yaml └── delivery.yaml # CI/CD config "Infrastructure as Code"
  61. 61. 66 ZALANDO IAM/OAUTH VIA CRD kind: PlatformCredentialsSet .. spec: application: my-app tokens: read-only: privileges: - com.zalando::foobar.read clients: employee: grant: authorization-code realm: users redirectUri: https://example.org/auth/callback Extensible Kubernetes API
  62. 62. 67 POSTGRES OPERATOR Application to manage PostgreSQL clusters on Kubernetes >700 clusters running on Kubernetes github.com/zalando/postgres-operator
  63. 63. Elasticsearch in Kubernetes Elasticsearch 2.500 vCPUs 1 TB RAM github.com/zalando-incubator/es-operator/
  64. 64. 69 SUMMARY • Application Bootstrapping • Git as source of truth and UI • 4-eyes principle for master/production • Extensible Kubernetes API as primary interface • OAuth/IAM credentials • PostgreSQL • CloudFormation for proprietary AWS services
  65. 65. 70 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate https://srcco.de/posts/accelerate-software-delivery-performance.html
  66. 66. 71 CONTAINERS From "Accelerate: The Science of Lean Software and DevOps"
  67. 67. 72 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate ≙ Commit to Prod ≙ Deploys/week/dev ≙ MTRS from incidents ≙ n/a
  68. 68. “.. means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience.” - ThoughtWorks Technology Radar
  69. 69. 74 PRODUCT MINDSET ... means we are developing all our products for the best customer experience and biggest business impact while all actions are aligned with our strategy.
  70. 70. 76 DEVELOPER SATISFACTION
  71. 71. 77 DOCUMENTATION "Documentation is hard to find" "Documentation is not comprehensive enough" "Remove unnecessary complexity and obstacles." "Get the documentation up to date and prepare use cases" "More and more clear documentation" "More detailed docs, example repos with more complicated deployments."
  72. 72. 78 DOCUMENTATION • Restructure following www.divio.com/en/blog/documentation/ • Concepts • How Tos • Tutorials • Reference • Global Search • Weekly Health Check: Support → Documentation
  73. 73. 80 NEWSLETTER "You can now.." • You can now benefit from the most recent Kubernetes 1.12 features, e.g. .. • You can now analyse your Kotlin project with SonarQube and upload your Scala code coverage report to SonarQube
  74. 74. 81 SIGNAL: ISSUE UPVOTES
  75. 75. 82 TESTIMONIALS “Useful information, good level of details and pleasant to read. It's one of the few newsletters that I took time to read entirely :)” - a reader, July 2018
  76. 76. 83 TESTIMONIALS “So, thank you, Team Automata, for listening to our community, taking our upvotes in consideration when developing new solutions and building every day 'the first CI that doesn't suck'.” - a user, October 2018
  77. 77. 84 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  78. 78. 85 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  79. 79. 86 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  80. 80. 87 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  81. 81. 88 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  82. 82. 89 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  83. 83. 90 DevOps is normal
  84. 84. 91 DevOps is normal
  85. 85. 92 You build it, you run it is normal
  86. 86. 93 WAIT A SECOND.. What about my job title?
  87. 87. 94 "DEVOPS ENGINEER" AS A JOB TITLE
  88. 88. 95 T-SHAPED ENGINEER "Generalizing specialist" Breadth of Knowledge TechnicalDepth
  89. 89. 96 DevOps Engineer Software Engineer / Software Developer (T-Shaped)
  90. 90. 97
  91. 91. 98 OPEN SOURCE & MORE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws Skipper HTTP Router & Ingress controller github.com/zalando/skipper External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando-incubator/postgres-operator More Zalando Tech Talks github.com/zalando/public-presentations
  92. 92. QUESTIONS? HENNING JACOBS HEAD OF DEVELOPER PRODUCTIVITY henning@zalando.de @try_except_ Illustrations by @01k

×