A few months ago, Vungle’s infrastructure was showing its age. As the company moved more toward microservices and needing globaly distributed infrastructure, our old approach of deploying a single app to a group of Ubuntu machines with Chef (either with autoscaling or manually) was starting to become a bottleneck. We were also worried that we were not utilizing our server resources well. We were already using Docker to streamline our development environments and CI systems, so moving production to a Docker-based system seemed like an obvious choice. After evaluating the options (Kubernetes, Mesos, Fleet, etc), we decided to go with Kubernetes on CoreOS.
This talk will focus on the technical decisions we made about our Kubernetes infrastructure to allow us to scale all over the globe, some of the issues we faced and how we worked around them, and the benefits we have seen.
Setting up clusters in VPCs using CloudFormation
Moving from legacy infrastructure
Exposing services to the outside world
Making complex http routing easily configurable by services
Communication between clusters
Limitations in AWS support
Integration into Deployment process
Vungle has benefitted greatly by embracing containers as the basic method for packaging services, and Kubernetes has really allowed us to become container-native all the way into production. It’s a lot of work to get it right, but putting in the effort is really paying off.