2013-cloudconnect-OpenStack@BT

463 views

Published on

A 15 minute talk I gave at Cloud Connect Chicago 2013 about how and why we use OpenStack at BrightTag

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
463
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

2013-cloudconnect-OpenStack@BT

  1. 1. OpenStack at BrightTag Joshua Buss – Cloud Connect 2013 ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  2. 2. Intro Me ● DevOps since before it was called “Dev Ops” ● Orbitz – NOC/SOC ● Graphite specialist ● High-frequency trading ● Capacity planning, DBA, *nix admin, HW geek ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  3. 3. Intro BrightTag ● World‟s leading Tag Management System ● Globally distributed on AWS ● Server-side tag firing through the cloud ● FUSE profile matching and offline processing ● Next-generation data management ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  4. 4. Motivation Lay of the land Four environments: • Local (each developer‟s laptop) • Development (in-office only) • Staging (publicly available, in Rackspace) • Production (AWS) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  5. 5. Motivation The problem • Production was on an EOL‟d debian • Staging and dev used same distro • Each region had separate puppetmaster • Local is (mostly) OSX (brew; “btiab”) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  6. 6. Motivation The fix • Moved production to ubuntu 11.10 • Centralized puppet repo • Had to rebuild dev and staging, too ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  7. 7. Motivation The opportunity • New dev environment? • Let‟s make it more like production! • Time to pick private cloud software ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  8. 8. Process The goals • As similar to Amazon as possible • Open source, active community • libcloud compatible • Highly-available • Web interface ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  9. 9. Process The choices • Ganeti • VMWare • CloudStack • Eucalyptus • OpenStack ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  10. 10. Process The choices • Ganeti (no web interface) • VMWare ($) • CloudStack (Chaotic at the time) • Eucalyptus (Tried it, didn‟t work out) • OpenStack (Tried it, worked great) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  11. 11. Process The work • Found some cheap servers on eBay • Put Ubuntu 12.04 server on them • Used our new puppet configs to add users, etc. • Followed the docs ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  12. 12. Process The work • Found some cheap servers on eBay • Put Ubuntu 12.04 server on them • Used our new puppet configs to add users, etc. • Followed the docs • Yeah, right. ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  13. 13. Process The work • Took about a month to learn: • Terminology • How all the pieces fit together • Highly available networking support • Routing and ARP tables • Thank you, IRC ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  14. 14. Result The finished product • Hey look, you can use this nice UI • ...or libcloud • ...and our single puppet master • ...and the result is just like our production instances! ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  15. 15. Result Open Stack was just the start • Dev environment now has over 50 VMs on 15 servers • Standardize all the environments! • ...even some new CI environments • ...and our local environments (virtualbox) • Jenkins is integral in our continuous deployments ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  16. 16. Thank you! ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  17. 17. Extras BT Infrastructure • Many small, HTTP webapps working together • Key off „server groups‟ to define each environment • “data01we2” - Server group “data” in the “we2” region "roles": {"we2": {"data": ["dataserve", "auxserve"] } … • Roles define which puppet modules we install ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  18. 18. Extras Zerg • libcloud + flask • Provisioning, real-time CMDB, generated configs • HAProxy, Nagios, Cassandra configs • “Manifest” - JSON that include IP Addresses • Use manifest to know where to deploy each app • Use zerg for DNS as well (updating /etc/hosts) ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  19. 19. Extras Puppet • Source-controlled through multiple branches • Puppet master has each branch checked out into a different directory • Branches are pulled automatically • Each branch has a matching puppet „environment‟ name • /etc/puppet/puppet.conf “environment” determines the client • All environments, including our guest VMs, default to master ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG
  20. 20. Extras Zero-downtime deploys • /bthc of each webapp determines healthiness of app • Fabric puts each app into maintenance mode before stopping • App stops taking traffic, then gets restarted with new code • New app doesn‟t respond with „healthy‟ until its actually ready for traffic • When there are a lot of an app, we parallelize the deploy ONE FOR ALL THE ADVANTAGES OF WORKING WITH BRIGHTTAG

×