Extreme Pipelines
•Download as PPTX, PDF•
0 likes•398 views
SpringOne Platform 2018 Extreme Pipelines William Marchlewski, Mastercard; Cory Jett, Pivotal
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Extreme Pipelines
Similar to Extreme Pipelines (20)
More from VMware Tanzu
More from VMware Tanzu (20)
Recently uploaded
Recently uploaded (20)
Extreme Pipelines
- 1. Extreme Pipelines Bill Marchlewski - Mastercard Cory Jett - Pivotal
- 2. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Why pipeline your Infrastructure? • Consistency • Within and between environments • Environment Promotion • Automated burn-in and promotion • Auditing • Who, what, when, where, and why • Change control • Clear, documented changes. Controlled rollout. • Do more with less • Team of 8 operators supporting PCF on Mastercard • Lazyness? • Let machines do the work. Have dinner at home. Sleep in on the weekends. 2
- 3. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Unit testing • Pipeline pull requests are tested via homegrown testing suite • New (or modified) pipelines are committed along with expected output (using test or “stubbed” data) • Test data is sourced from a successful concourse pipeline i.e. `fly –t target get-pipeline` • This test data is used as test success criteria to ensure that a PR doesn’t “break the build” • Feature branches are automatically built in Concourse and build status reported in Bitbucket
- 4. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Install We use two Concourse teams per foundation. Everything starts with a clean slate in vSphere and a Service Account • Install team • Installs Opsman, BOSH director, and the basics needed to get the platform up • Operations team (manages everything else) • Deploys all tiles, upgrades, and build packs • Configures load balancers • Performs platform validation and smoke test on a timely basis • Runs general maintenance • BOSH Prometheus deployment • Hashicorp Vault deployment Pivotal Cloud Foundry 1 of 5000
- 5. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Install OM/BOSH 5
- 6. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Install tiles 6 Dirty jokes redacted…
- 7. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Upgrades • Pipeline is tied to PivNet • Pulls down latest product version based on `ver` regex • .pivotal file is stored in local S3 bucket • Pipeline defines • Included environments (whitelist) • Excluded environments (blacklist) • Dependencies (i.e. PCF Metrics depends on ERT)
- 8. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Upgrades Template file .json: • Created from tile properties • Pulled from OM after a successful manual install for minor releases in Sandbox • “Templatized” so it can be merged with environment specific properties via Spruce • Uploads new product, applies configuration, and applies the changes
- 9. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Promotion • New updates are automatically applied in SND and promoted up to the next environment after a 2 week burn-in • SND -> DEV -> STG -> PRD • 4 weeks from SND to PRD • Promotion can be expedited if needed to move changes faster (such as a critical stemcell update)
- 10. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ PCF Onboarding Onboarding is done via pull request and pipeline • Users submit a pull request for a new Org containing • Name of the Org and Space(s) • Quota details • Permissions • Org and Space permissions mapped to LDAP group(s) • Jenkins credential details • Contact information • If a sandbox space is needed (a special space where `cf push` works)
- 11. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ PCF Onboarding • PRs are automatically scanned and status reported in Bitbucket • Checks syntax • Duplicate Org/Space naming • LDAP verification • Resource limits. Auto resource bump when close to max • Once the pull request is approved and the Org Management pipeline runs • Orgs and Spaces will be created and permissions applied as defined within the request. • The account will be added on the next cycle of the pipeline (+/- 5 minutes) • A welcome email will be sent to the contacts if a new Org is added
- 12. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Platform Health • Is the app up? • Pushes an app and makes sure it is on- line • Validate Base • Checks that all PCF services are on-line • Login to OM, login to PCF, check bosh capacity... • Validate Dependencies • vSphere resources (datastores), DNS • Validate Services • Pushes a small app • Binds the app to a service for testing • Unbinds and deletes the app • Check Connectivity • Validates connectivity to external dependencies • BOSH cleanup
- 13. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Pipeline Health “The Dashboard” • Pulls from the monitors in all environments. • Checks pipelines for errors • Checks for failed upgrades • Hover over a box to see what the process is, click to get to the pipeline. • Alerts • Concourse sends notifications to team chat • Critical alerts sent to alerting system to page on-call resource
- 14. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ Deep thoughts 14 • Automate early • Retroactively pipelining is way harder than doing the work up front • Pipelines are not a “magic bullet” • They require continued dedication and discipline to be successful • There is an upfront cost in engineering effort to build solid pipelines • This can be steep and initially hard to justify • The ROI after building well oiled pipelines is worth it • Avoid the temptation • Sometimes its easier to do something manually vs. spending the time to automate it • Don’t be that guy/gal • Think long and hard about the time investment now vs. the ongoing time required to account for that snowflake you just made.
- 15. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/ What’s next? 15 • PKS • Automate the deployment, configuration, and management of Pivotal Container Services • NSX-T • Automate the complete buildout and configuration of NSX-T for Software Defined Networking • Automate the deployment of PAS and PKS on top of NSX-T
- 16. > Stay Connected. @stlguitarist (Bill) cjett@pivotal.io (Cory) #springone@s1p
Editor's Notes
- Cory Mastercard customer for about 2 years Using a number of pivotal technologies Day 1 priority to build a best in class platform team with a diverse skill dedicated to platform architecture and operations Why automation? Probably preaching to the choir on automation We’ve been doing this for years, benefits are well understood at this point 3 global datacenters 8 folks on the core platform team supporting thousands of developers
- Cory
- Bill M. At this time we have determined that the vSphere infrastructure, firewall ports, the four networks and all prerequisites are in place. We have the clusters for the AZs, in our case ready. One logs into the Env-Install team in Concourse and kicks off the pipeline. The install pipeline lays down the basic OpsMan and basic PCF components with all required fields in Opsman and director defined. the opsmgr.yml and director.yml. Once this is complete we can log into opsmanger Next we log into the Operations team in concourse and kick off the rest starting PAS and working on down based on dependency. Once PCF is installed we are far from done. Our pipelines are unpaused to upgrade opsman, install the Build packs, Prometheus on BOSH, Vault, configure the F5 load balancers. Turn on platform validation smoke test, a general maintenance pipeline and the one that runs org management.
- billm
- billm
- billm We with the the S3 pull down due to possible download issues of which we encountered early on. This also allows us to move the .pivotal files between buckets based on environments. the
- For major and minor releases, we do a manual install in our sandbox environment. Once installed and passing all apply settings, we pull the properties and add the changed (if any) to the new template file. We also add any environmental variables into the foundation specific tiles.yml files. Once all the config files are modified, we pull the manual install. Form there we do a pull request to and have the code go through it’s normal promotion and install processes.
- Automatic updates (incremental versions) are done automaticly. (they’re suppose to follow Pivotal’s guidelines for defining a release). We’ve had cases with incremental releases have had changes that won’t run through requiring some tweaks. I will say we’ve only encountered this through third party tiles of which I won’t mentions. if the tile installs in Sandbox automaticly we get notifed. if all is well after one week it moves to Dev, a week later to stage, the in two weeks cycles through our Prod environments.
- Once the users are approved to use PCF, they request an LDAP group. Once those requirements are met, the rest flows via pipeline. The users modify a name.org.yml file and create a pull request.
- billm The user puts in a pull request for the .yml. Before it can be approved it too much pass run the acceptance test. Once it passes it and the pull request is approved, the orgs and spaces will be added when the next cycle of the org management pipeline runs.
- billm This is runs many test. This pileline check the entire foundation for critical componets and full functioniality. Every tile and service added gets put into the smoke test. Checks connectivity to all critical networks and services. Validates dependencies. vSphere resources, Check utilization check the status of the VMs running in the cluster. if any PCF Components are running hot we get alerted. This along with Promtheus and PCF Heatlthwatch allow us to tweak things as needed. Example would be add additional Doppler VMs, Elastic Search Nodes or change the size of the VMs themselves, cores, memory etc.
- billm This is our on call tool that pull data from all operations pipelines described prior. We can see what is going on in Pivnet, all foundations, all upgrades, failed validation checks (DEMO??? if time permits.
- Cory Pipeline first, do it right Pipelines are technically complex and hard
- Cory PKS - Deployment of PKS, provisioning of K8S clusters, and providing stardardized/abstracted onboarding mechanisms for consumers NSX-T from hypervisor to SDN using Terraform
- Cory