Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCon Berlin 2019

737 views

Published on

While the adoption of DevOps makes teams move faster with reduced dependency on central operations, it can constrain teams who lack the skills to self-manage the full application and infrastructure stack. The way to overcome this challenge is creating an internal platform and treating it as a world-class product offering. “Applying product management to internal platforms means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience”, via ThoughtWorks Technology Radar. In this talk, we will walk you through how Zalando adopted a customer-first mindset with regards to its developer tooling. We will show the effect on developer satisfaction when internal platforms are given the same respect as external product offerings. We will tell our story on how we moved from a classical infrastructure team to a product mindset with strong focus on building a world-class developer experience. We will share both our learnings and challenges going through this transition, and the impact it has on the daily life of our customers (developers).

Published in: Technology
  • Be the first to comment

Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCon Berlin 2019

  1. 1. Why we don't use the term DevOps 2019-06-12 HENNING JACOBS @try_except_ MOSTAFA NAGEEB @MostafaNageeb
  2. 2. 2 ZALANDO AT A GLANCE ~ 5.4billion EUR revenue 2018 > 250 million visits per month > 15.000 employees in Europe > 79% of visits via mobile devices > 26 million active customers > 300.000 product choices ~ 2.000 brands 17 countries
  3. 3. 3 THE BIRTH OF DEVOPS - 2009 “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” DevOps Days Belgium https://devops.com/the-origins-of-devops-whats-in-a-name/
  4. 4. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
  5. 5. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
  6. 6. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
  7. 7. 7 THE PHOENIX PROJECT - 2013 "The Three Ways" 1. Systems Thinking 2. Amplify Feedback Loops 3. Culture of Continual Experimentation And Learning
  8. 8. 8 WHAT HAPPENED NEXT
  9. 9. 9 CARGO CULTING
  10. 10. 10 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  11. 11. 11 DevOps is normal
  12. 12. 12
  13. 13. 13 WHAT HAPPENED NEXT
  14. 14. 14 WHAT HAPPENED NEXT
  15. 15. 15 DEV + OPS ⇒ DEVOPS Devs "DevOps" Team Ops
  16. 16. 16 DEV + OPS ⇒ ENGINEERING Devs "DevOps" Team Ops Engineering Team
  17. 17. 17 PRODUCT DEVELOPMENT TEAMS "Great products emerge from empowered, fully cross-functional teams." - John Cutler, 2019 https://amplitude.com/blog/on-being-product-led
  18. 18. 18 DEVOPS IS It is a culture, in which people work together to improve the product delivery cycle.
  19. 19. 19 DEVOPS ANTI-PATTERNS Dev and Ops Silos DevOps Team Silo Dev Don't Need Ops DevOps as Tools Team Rebranded SysAdmin Ops Embedded in Dev Team web.devopstopologies.com
  20. 20. 20
  21. 21. 21 TEAM INTERACTION MODES • Collaboration: 2 teams working together • X-as-a-Service: 1 provides, 1 consumes • Facilitating: 1 team helps another Product teams need a family too - DevOpsDays Portugal
  22. 22. 22 A BRIEF HISTORY OF ZALANDO TECH
  23. 23. 23 2010 "Sysop-Test" "QA-Test"
  24. 24. 24 DEVOPS IS ABOUT SKIN IN THE GAME Absence of skin in the game: People who are isolated from the impacts of their decisions do not learn. They remain captive to their erroneous ideas about how the world works.
  25. 25. 25 2013: SELF SERVICE
  26. 26. 26 2015: RADICAL AGILITY AWS STUPS DOCKER DEPLOY SSH ACCESS AUDIT REPORTS FULL AWS ACCESS Teams have admin access & full responsibility
  27. 27. 27 2015: ISOLATED AWS ACCOUNTS Internet *.abc.example.org *.xyz.example.org Team ABC Team XYZ EC2EC2 ELBELB EC2
  28. 28. 28 Platform > 1100 developers > 200 development teams 2019
  29. 29. 29 YOU BUILD IT, YOU RUN IT The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. - A Conversation with Werner Vogels, ACM Queue, 2006
  30. 30. 30 ON-CALL: YOU OWN IT, YOU RUN IT When things are broken, we want people with the best context trying to fix things. - Blake Scrivener, Netflix SRE Manager
  31. 31. 31 "SKIN IN THE GAME"
  32. 32. 32 DEVELOPER JOURNEY Consistent story that models all aspects of SW dev
  33. 33. 33 Developer Journey
  34. 34. 34 Developer Journey Correctness Compliance GDPR Security Cost Efficiency 24x7 On Call Governance Resilience Capacity ...
  35. 35. 35 DEVELOPER PRODUCTIVITY Code Build Test Deploy OperateSetup Cloud Native Application Runtime
  36. 36. 36 CLOUD NATIVE .. uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. Cloud native technologies enable software developers to build great products faster. - https://www.cncf.io/
  37. 37. 37 CONTAINERS END-TO-END Code Build Test Deploy OperateSetup Cloud Native Application Runtime
  38. 38. 38 CONTAINERS
  39. 39. 39 CONTAINERS
  40. 40. 41 PLAN & SETUP
  41. 41. 42 Plan Stories Rules of Play Tech Radar
  42. 42. 44 Setup Application Bootstrapping
  43. 43. 47 BUILD & TEST
  44. 44. 48 CDPGit code push CONTINUOUS DELIVERY PLATFORM: BUILD
  45. 45. 50 DEPLOY
  46. 46. 51 Deploy Kubernetes
  47. 47. 52 DEPLOYMENT CONFIGURATION ├── deploy/apply │ ├── deployment.yaml │ ├── credentials.yaml # Zalando IAM │ ├── ingress.yaml │ └── service.yaml └── delivery.yaml # Zalando CI/CD
  48. 48. 53 INGRESS.YAML kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80
  49. 49. 54 TEMPLATING: MUSTACHE kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "{{{APPLICATION}}}.example.org" http: paths: - backend: serviceName: "{{{APPLICATION}}}" servicePort: 80
  50. 50. 55 CONTINUOUS DELIVERY PLATFORM
  51. 51. 56 CDP: DEPLOY "glorified kubectl apply"
  52. 52. 57 CDP: OPTIONAL APPROVAL
  53. 53. 58 STACKSET: TRAFFIC SWITCHING github.com/zalando-incubator/stackset-controller
  54. 54. 59 STACKSET CRD kind: StackSet ... spec: ingress: hosts: ["foo.example.org"] backendPort: 8080 stackLifecycle: scaledownTTLSeconds: 1800 limit: 5 stackTemplate: spec: podTemplate: ... github.com/zalando-incubator/stackset-controller
  55. 55. 60 TRAFFIC SWITCHING STEPS IN CDP github.com/zalando-incubator/stackset-controller
  56. 56. 61 EMERGENCY ACCESS SERVICE Emergency access by referencing Incident zkubectl cluster-access request --emergency -i INC REASON Privileged production access via 4-eyes zkubectl cluster-access request REASON zkubectl cluster-access approve USERNAME
  57. 57. 62 INTEGRATIONS
  58. 58. 63 CLOUD FORMATION VIA CI/CD ├── deploy/apply │ ├── deployment.yaml # Kubernetes │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml │ ├── kube-secret.yaml │ └── kube-service.yaml └── delivery.yaml # CI/CD config "Infrastructure as Code"
  59. 59. 64 ZALANDO IAM/OAUTH VIA CRD kind: PlatformCredentialsSet .. spec: application: my-app tokens: read-only: privileges: - com.zalando::foobar.read clients: employee: grant: authorization-code realm: users redirectUri: https://example.org/auth/callback Extensible Kubernetes API
  60. 60. 65 POSTGRES OPERATOR Application to manage PostgreSQL clusters on Kubernetes >700 clusters running on Kubernetes github.com/zalando/postgres-operator
  61. 61. Elasticsearch in Kubernetes Elasticsearch 2.500 vCPUs 1 TB RAM github.com/zalando-incubator/es-operator/
  62. 62. 67 SUMMARY • Application Bootstrapping • Git as source of truth and UI • 4-eyes principle for master/production • Extensible Kubernetes API as primary interface • OAuth/IAM credentials • PostgreSQL • CloudFormation for proprietary AWS services
  63. 63. 68 Platform as a Product
  64. 64. “.. means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience.” - ThoughtWorks Technology Radar
  65. 65. 70 PRODUCT MINDSET ... means we are developing all our products for the best customer experience and biggest business impact while all actions are aligned with our strategy.
  66. 66. 71 KNOW YOUR AUDIENCE
  67. 67. 72 ● Be Prepared. ● Talk to both users and NON users. ● Ask about problems, not solutions. ● Open ended questions. TALK TO USERS
  68. 68. 73 IDENTIFY COMMON PROBLEMS "Documentation is hard to find" "Documentation is not comprehensive enough" "Remove unnecessary complexity and obstacles." "Get the documentation up to date and prepare use cases" "More and more clear documentation" "More detailed docs, example repos with more complicated deployments."
  69. 69. 74 TRANSPARENTLY COMMUNICATE WHAT YOU ARE SOLVING
  70. 70. 75 SIGNAL: ISSUE UPVOTES
  71. 71. 76 ENGAGE CUSTOMERS EARLY ON We are working on speeding up builds by providing basic overlays for the most used languages/frameworks so you don't have to download the internet every time you are building your app. This is the approach and the documentation (not live yet), please add your comments to guide us in the right direction.
  72. 72. 77 SUPPORT IS A FEATURE
  73. 73. 78 DOCUMENTATION • Restructure following www.divio.com/en/blog/documentation/ • Concepts • How Tos • Tutorials • Reference • Global Search • Weekly Health Check: Support → Documentation
  74. 74. 80 COMMUNICATION Guidelines • Target announcements • Use the right tags • Timing
  75. 75. 81 NEWSLETTER "You can now.." • You can now upload your Scala code coverage report to SonarQube "This enables you to.." • This enables you to browse your code quality results and track it over time
  76. 76. 82 “Useful information, good level of details and pleasant to read. It's one of the few newsletters that I took time to read entirely :)” - a reader, July 2018
  77. 77. 83 EMPOWER YOUR TEAM
  78. 78. 84 WITH GREATER POWER COMES GREATER RESPONSIBILITY
  79. 79. 85 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate srcco.de/posts/accelerate-software-delivery-performance.html
  80. 80. 86 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate • Commit to Prod • Deploys/week/dev • MTRS from incidents • N/A
  81. 81. 87 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate • Commit to Prod • Deploys/week/dev • MTRS from incidents • N/A
  82. 82. 89 DEVELOPER SATISFACTION
  83. 83. 90 INSIGHTS THAT GUIDE THE PRODUCT “Kubernetes/CDP deploys are too slow - ours takes over 10 minutes, mainly due to having to download SBT dependencies every time.”
  84. 84. 91 INSIGHTS THAT GUIDE THE PRODUCT “CDP should somehow cache builds or something like that. The current approach forces everyone to re-invent the cache wheel for their own projects - or face a crazy slow build.”
  85. 85. 92 INSIGHTS THAT GUIDE THE PRODUCT “Kubernetes/CDP deploys are too slow - ours takes over 10 minutes, mainly due to having to download SBT dependencies every time.”
  86. 86. 93
  87. 87. 94 TESTIMONIALS “So, thank you, Team Automata, for listening to our community, taking our upvotes in consideration when developing new solutions and building every day 'the first CI that doesn't suck'.” - a user, October 2018
  88. 88. 95 TESTIMONIALS “Thank you very much for improving the color scheme of the UI. It really helps me with my color blindness. It used to be hard to distinguish between statuses previously.” - a user, February 2019
  89. 89. 96
  90. 90. 97 Wrap Up
  91. 91. 98 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  92. 92. 99 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  93. 93. 100 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  94. 94. 101 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  95. 95. 102 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  96. 96. 103 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
  97. 97. 104 DevOps is normal
  98. 98. 105 DevOps is normal
  99. 99. 106 You build it, you run it is normal
  100. 100. 107 WAIT A SECOND.. What about my job title?
  101. 101. 108 "DEVOPS ENGINEER" AS A JOB TITLE
  102. 102. 109 T-SHAPED ENGINEER "Generalizing specialist" Breadth of Knowledge TechnicalDepth
  103. 103. 110 DevOps Engineer Software Engineer / Software Developer (T-Shaped)
  104. 104. 111 ON THE DEVOPS JOB TITLE "Everyone’s a devops engineer these days. I’m a devops engineer, my wife’s a devops engineer, even my dog’s a devops engineer" - James Betteley, 2014 devopsnet.com/2014/08/13/the-devops-team-myth/
  105. 105. 112
  106. 106. 113 OPEN SOURCE & MORE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws Skipper HTTP Router & Ingress controller github.com/zalando/skipper External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando-incubator/postgres-operator More Zalando Tech Talks github.com/zalando/public-presentations
  107. 107. QUESTIONS? HENNING JACOBS @try_except_ MOSTAFA NAGEEB @MostafaNageeb Illustrations by @01k

×