Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Leveraging Automation to Scale


Published on

Leveraging Automation to Scale was originally presented at Lookout's Scaling for Mobile event on July 25, 2013. Jim Hopp is an Operations Architectr at Lookout, Inc. Jim's talk focused on moving quickly with a small organization—and using automation to do so. Lookout has grown immensely in the last year. We've doubled the size of the company—added more than 80 engineers to the team, support 45+ million users, have over 1000 machines in production, see over 125,000 QPS and more than 2.6 billion requests/month. Our analysts use Hadoop, Hive, and MySQL to interactively manipulate multibillion row tables. With that, there are bound to be some growing pains and lessons learned.

Published in: Technology, Business
  • Be the first to comment

Leveraging Automation to Scale

  2. 2. HOW TO KEEP UP? • Increasing number of services • And desire for faster cycle time • Increasing number of machines • Frequent deploys (and getting more frequent) • Engineering growing much faster than Ops
  3. 3. (TESTING?!) • Important that provisioning chgs follow modern dev pipeline • rspec (chefspec) for chef • minitest for chef’d nodes • code reviews • deploys from jenkins • continually re-provision bare-metal test nodes
  4. 4. GET OTHERS TO DO THE WORK • We want engineers to write the chef cookbooks and cap config files for their service • We want them to use the production tools during the development cycle • Originally built vagrant-based tools. Worked well but fragile and didn’t scale past 2-3 VMs • Developed TestLab for managing Linux containers. Works great! zpatten/testlab
  5. 5. MAKE IT SIMPLE TO USE • Built Chef LWRPs to simplify configuration • This is all that’s necessary for a normal service: • LWRPs for resque, firewall, SSL certs, etc. • Libraries for finding IPs, data bags within environments, etc. • Office hours, pairing lookout_passenger_app 'my_service' do rvm_gemset 'ruby-1.9.3-p448@my_service' end
  6. 6. DEPLOYS • chef-repo deployed to prod on merge to master by jenkins • Deploy HEAD • For services, moving to developer deploy • They know their service better than Ops • Still figuring out migrations • Tooling for roll-back
  7. 7. Keep in touch with @lookout /mylookout