LEVERAGING
AUTOMATION TO SCALE
JIM HOPP, LOOKOUT, INC.
HOW TO KEEP UP?
• Increasing number of services
• And desire for faster cycle time
• Increasing number of machines
• Frequ...
(TESTING?!)
• Important that provisioning chgs follow modern dev pipeline
• rspec (chefspec) for chef
• minitest for chef’...
GET OTHERS TO DO THE WORK
• We want engineers to write the chef
cookbooks and cap config files for their service
• We want t...
MAKE IT SIMPLE TO USE
• Built Chef LWRPs to simplify configuration
• This is all that’s necessary for a normal service:
• L...
DEPLOYS
• chef-repo deployed to prod on merge to master by
jenkins
• Deploy HEAD
• For services, moving to developer deplo...
Keep in touch with
@lookout
/mylookout
blog.lookout.com
contact@lookout.com
http://bit.ly/scaling-for-mobile
Upcoming SlideShare
Loading in …5
×

Leveraging Automation to Scale

815
-1

Published on

Leveraging Automation to Scale was originally presented at Lookout's Scaling for Mobile event on July 25, 2013. Jim Hopp is an Operations Architectr at Lookout, Inc. Jim's talk focused on moving quickly with a small organization—and using automation to do so. Lookout has grown immensely in the last year. We've doubled the size of the company—added more than 80 engineers to the team, support 45+ million users, have over 1000 machines in production, see over 125,000 QPS and more than 2.6 billion requests/month. Our analysts use Hadoop, Hive, and MySQL to interactively manipulate multibillion row tables. With that, there are bound to be some growing pains and lessons learned.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
815
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Leveraging Automation to Scale

  1. 1. LEVERAGING AUTOMATION TO SCALE JIM HOPP, LOOKOUT, INC.
  2. 2. HOW TO KEEP UP? • Increasing number of services • And desire for faster cycle time • Increasing number of machines • Frequent deploys (and getting more frequent) • Engineering growing much faster than Ops
  3. 3. (TESTING?!) • Important that provisioning chgs follow modern dev pipeline • rspec (chefspec) for chef • minitest for chef’d nodes • code reviews • deploys from jenkins • continually re-provision bare-metal test nodes
  4. 4. GET OTHERS TO DO THE WORK • We want engineers to write the chef cookbooks and cap config files for their service • We want them to use the production tools during the development cycle • Originally built vagrant-based tools. Worked well but fragile and didn’t scale past 2-3 VMs • Developed TestLab for managing Linux containers. Works great! https://github.com/ zpatten/testlab
  5. 5. MAKE IT SIMPLE TO USE • Built Chef LWRPs to simplify configuration • This is all that’s necessary for a normal service: • LWRPs for resque, firewall, SSL certs, etc. • Libraries for finding IPs, data bags within environments, etc. • Office hours, pairing lookout_passenger_app 'my_service' do rvm_gemset 'ruby-1.9.3-p448@my_service' end
  6. 6. DEPLOYS • chef-repo deployed to prod on merge to master by jenkins • Deploy HEAD • For services, moving to developer deploy • They know their service better than Ops • Still figuring out migrations • Tooling for roll-back
  7. 7. Keep in touch with @lookout /mylookout blog.lookout.com contact@lookout.com http://bit.ly/scaling-for-mobile
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×