Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Draupnir: A Story About Managing Concourse in the Enterprise

112 views

Published on

SpringOne Platform 2018
Draupnir: A Story About Managing Concourse in the Enterprise
Speakers: Evan Brown Senior Software Engineer, Google Topher Bullock Staff Software Engineer, Pivotal Nick Penston Cloud Application Deployment - Lead, Fidelity Investments

Published in: Software
  • Be the first to comment

  • Be the first to like this

Draupnir: A Story About Managing Concourse in the Enterprise

  1. 1. A Story about Managing Concourse in the Enterprise Draupnir
  2. 2. Topher Bullock, Pivotal cbullock@pivotal.io @topherbullock
  3. 3. A Tale of Concourse In the Enterprise
  4. 4. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse in The Enterprise Platform Operators: - Using Concourse to deploy and manage PCF Foundations - Need to operate Concourse as well as PCF - Often bootsrapping many Concourse deployments App Developers: - Concourse pipelines are the path to production - Rely on the operator to keep the platform up Increasingly valuable and critical to Platform Operators and App Developers
  5. 5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ In Norse mythology, Draupnir is a gold ring possessed by the god Odin with the ability to multiply itself. 365 ÷ 9 = 40.55 nights / year 40.55 x 8 = ~324.44 rings / year Every ninth night, eight new rings 'drip' from Draupnir, each one of the same size and weight as the original. Draup-nir??
  6. 6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
  7. 7. A Tale of Concourse Enterprise Operations at Scale
  8. 8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Platform Operators
  9. 9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
  10. 10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Day 0 Day n
  11. 11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Day n
  12. 12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App Developers
  13. 13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
  14. 14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
  15. 15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Day 1 Bootstrapping Day n Scaling
  16. 16. Evan Brown, Google evanbrown@google.com
  17. 17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 7 Concourse at
  18. 18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ In the beginning 1 8 Cloud Foundry
  19. 19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 9 Cloud Foundry Terraform Chef Ansible Puppet Salt Vault ELK
  20. 20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2 0
  21. 21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Creating Concourse Manually 2 1 BOSH
  22. 22. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Not enough automation
  23. 23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ No conventions for consistency
  24. 24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Concourse Pipeline 2 4
  25. 25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Too much automation
  26. 26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ You end up doing weird stuff
  27. 27. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ like adding a notch
  28. 28. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ BOSH Bootloader (bbl) A command line utility for standing up BOSH on an IaaS 2 8
  29. 29. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ BOSH Bootloader A command line utility for standing up BOSH on an IaaS 2 9 BOSHbbl
  30. 30. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Right mix of automation
  31. 31. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ and convention
  32. 32. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ no-notchsense
  33. 33. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Using the BOSH Bootloader 3 3 $ export BBL_IAAS=gcp $ export BBL_GCP_REGION=us-east1 $ export BBL_GCP_SERVICE_ACCOUNT_KEY=key.json $ bbl init $ bbl up $ bosh deploy concourse $ git init && git add && git commit
  34. 34. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Using the BOSH Bootloader 3 4 $ export BBL_IAAS=gcp $ export BBL_GCP_REGION=us-east1 $ export BBL_GCP_SERVICE_ACCOUNT_KEY=key.json $ bbl init $ bbl up $ bosh deploy concourse $ git init && git add && git commit WE’RE DOING IT LIVE
  35. 35. Nick Penston, Fidelity Investments
  36. 36. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Empathizing With Operator Toil
  37. 37. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pivotal Internal Large Concourse Deployment 80 + teams 3 ATCs 22 workers Lots of active pipelines Following Google Site Reliability Engineering (SRE) playbook. Wings
  38. 38. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
  39. 39. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Roadmap
  40. 40. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Proposal: Metrics / Tools for Operators running large-scale Concourse #2249 Concourse Helm Chart: Improve & Bless Concourse (Garden) Workers and ATCs on K8s #1731 Compatibility matrix For interoperability with Pivotal Releases ( UAA, Credhub ) .Operations. Concourse for PCF Resources Pivotal Maintained resources packaged with Concourse for PCF .Concourse for PCF.
  41. 41. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Multi-tenant runtime scalability Balancing jobs across workers, resource limits, untrusted MT #2271 Distribute container/volume garbage- collection across workers #1959 .Runtime. Improving Concourse’s “Runtime” as a feature has been hard to tie to end-users in the past Focus on the outcomes we want to achieve for customers has helped to drive out new features in this track Resilency, Efficiency and Scalability of the runtime all help customer Platform Operators
  42. 42. Questions
  43. 43. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation. 4 6

×