Continuous Delivery at Gogo
With Spinnaker and Foremast
Doug Campbell
Twitter: @ndougops
Github: @ndcampbell
Overview
DevOps at Gogo
Continuous Delivery Workflow
Spinnaker and Foremast demos
DevOps at Gogo
Developers developers developers
Cattle not cats (immutable infra)
Evangelize and educate
Document everything
Our Workflow
The Numbers
Git Repos: ~1,080
Metrics per day: ~358,000
Apps in pipeline: ~150
Deployments per day: ~70
Our Tools
Jenkins
Gitlab
Terraform
Packer
Spinnaker
Foremast
Python
Spinnaker
Why Spinnaker?
● Open source
● Multi-region and multi-cloud deployments
● Large dedicated community
● Central interface to all deployments
● Great API
Spinnaker Architecture
Spinnaker Demo
Foremast
Why Foremast?
Spinnaker pipelines as code
No manual creation of pipelines in the UI
Reproducible and versioned pipelines
Highly customizable pipelines via Jinja2 templates
Handles AWS resources not supported in Spinnaker
IAM, S3, Lambda
Foremast Demo
Thanks!
Twitter: @NDougOps
Github: @ndcampbell
tech.gogoair.com
github.com/gogoair/foremast

Continuous Delivery at Gogo with Spinnaker and Foremast

Editor's Notes

  • #2 Hello’s Introduction going to be talking about Continuous Delivery with Spinnaker and Foremast today.
  • #3 I’ll go over what DevOps means at Gogo our general workflow Most of presentation is Spinnaker and Foremast help us deliver applications quickly and consistently to AWS. The demos will be live, so just pray to the demo-gods for me that everything goes as planned.
  • #4 We try to follow these core tenets Developers Developers Developers. Everything we do is to try to help enable the developers deliver code easier, faster, and with less risk. Provide them with tooling Live by the “Cattle not Cats” analogy. Means that infrastructure is treated like Cattle. It is mostly identically, disposable, and easily replaceable. When it get sick you shoot it. Cats are special snowflakes, who you name, with unique personalities. When it gets sick you go to vet. Use immutable infrastructure. Everything defined as code. Evangelize and educate DevOps is not just one team Organization wide effort. Constantly evangelize the DevOps ideals Educate the developers on our tooling, processes, goals. Document everything. We want Developers to be in control of their stack Good documentaiton is Key We do not want to just become a deployment support team we want the users to be able to support themselves
  • #5 I stole these next two slides give a good visual representation of DevOps goal at Gogo. mission of DevOps at Gogo is to go from this: Silo’ed roles, too many deployment steps, too many people touching everything….
  • #6 To this. The developers being in control of their code, their monitoring their deployments. This leads to smaller, faster releases which are a lot easier to control, validate, and rollback if necessary.
  • #7 This is our basic deployment workflow. Developer pushes a code change and triggers a jenkins job. Jenkins builds the application and uploads an RPM artifact. Spinnaker monitors Jenkins and triggers based on success. Spinnaker gets the generated Artifact and makes an AMI. Spinnaker then setups up AWS infrastructure, the ELBs, security groups, roles, etc, Spinnaker creates Auto Scaling Groups with the generated AMI and spins it up. Once healthchecks pass, it spins down old clusters We then run Quality Engineering checks against the deployment The developer has visibility into this whole pipeline through Spinnaker and knows if a specific step or check has failed.
  • #8 Here are just some numbers on our current scale. These numbers are going up very quickly as we get more and more legacy applications to move over.
  • #9 I don’t think tools are as important as the process but everyone is always curious so I wanted to give some free advertisement to the awesome tools that we use.
  • #11 I will get to WHAT spinnaker is in the demo, but Why should you be interested? open source tool written by Netflix, Google, Microsoft and more. multi-region and multi-cloud. AWS, Google Compute Engine, Azure, Openstack, Kubernetes, and more. Avoid cloud lockin large very dedicated community. slack channel Companies like Google, Microsoft, Netflix, Pivital, Target, etc center management point for all deployments regardless of where it is. central cloud inventory as well good API. We have written a lot of tooling around Spinnaker and with the API you can make very custom workflows.
  • #12 Bunch of microservices Baking images, UI, authentication, communicating with the cloud, API entrypoint, pulling for jenkins, sending notifications. Redis required Cassandra is being fazed out, can use S3 or Redis for most things now. Looks complicated because it is They provide a prebuilt AMI for testing and is a great starting point. We have 4 cloud driver instances, it is the heaviest
  • #13 So what actually is spinnaker? 1. Login to spinnaker, show it supports SAML, Google Auth, Github auth 2. Show off the UI. Show that I can view all my cloud applications 3. Click on 429servicebsg, a real application in PROD 4. Show load balancers, instances, security groups. 5. Show pipeline, explain steps, show successes and failures 6. Walkthrough creating a new application 7. Show pipeline creation. Go over triggers, stages, manual judgement, etc. 8. Show manually creating ELBs, security groups.
  • #15 Spinnaker demo, some obvious questions may have came up How do you make the pipelines for 100+ applications? Manually? Can you automate creating the load balancers, security groups, and other? We made Foremast to help with those. No manual pipeline creation in the UI Easily reproducible Handles all AWS infrastructure Customizable pipelines to fit any need Open source and available. Good interest from the community.
  • #16 Show foremast.cfg, pipeline.json, application.json Run command and explain variables Show in spinnaker that the application was created show the generated pipeline Run the pipeline Show instance size Change it on the templates and rebuild Show new instance size Show Lambda pipelines and explain how it works Show documentation and getting started guide Plug Gitter and how we want to help