Puppet at Backstop
Another year of lessons
1
$ whoami
‣ Bill Weiss <bill@backstopsolutions.com>
‣ @BillWeiss
‣ Probably responsible for 90% of the badness in
this talk...
What’s this talk about?
aka, who cares?
3
4
Image via MemeCrunch. Original owned by the History Channel.
No, wait, that’s not it
(Come talk to me afterward about that)
5
‣ I’m here to talk about things that work (and don’t)
for us in our environment
‣ Hopefully most of it is useful to you
‣ ...
–You
“What the heck is Backstop?”
7
Super helpful, I know
8
SaaS for institutional
investment firms
‣ That’s it.
‣ You’re probably not our target audience
‣ If you are, awesome, let’s...
Some numbers
‣ ~195 Puppeted machines in production (US)
‣ ~350 total
‣ 2 datacenters, 2 AWS AZs, 1 office closet full of
m...
Logo owned by the CentOS project
Logo also owned by Oracle, of course
Logo owned by Oracle
Logo by David Heinemeier Hansso...
‣ Big application is J2EE on JBoss
‣ Lots of smaller bits (“services”) on Tomcat
‣ Second app is Ruby on Rails
12
How ‘bout that Puppet?
‣ Moved to GitHub 2013/11/18
‣ ~4100 commits since then (~15 commits/day!)
‣ 2700 from people outsi...
The move to GitHub was
awesome
‣ 4 years of Subversion before that had 29 authors
and 6700 commits
‣ Other things at play ...
! git workflow
‣ Org-owned repo, two branches
‣ master ➡ production
‣ office ➡ testing (pre-prod)
‣ Everyone forks from mast...
‣ Everyone has their own fork with work branches
‣ Branch names probably are Jira ticket numbers
‣ Everyone bases from mas...
This is why we have
so many contributors
17
‣ If you’re a software shop, you already have a
workflow. Do that
‣ Otherwise, this is working well for us
18
" git hooks
‣ We had a pretty good set of pre-commit hooks in
svn to test changes
‣ Github doesn’t do that #
‣ Git pre-com...
! Jenkins!
‣ We used to have this “deploy” job on the Puppet
masters as a cron job:
• (cd /etc/puppet && svn up)
‣ Nothing...
‣ Jenkins polls for changes to either branch
‣ Runs a deploy job that grabs repo, runs
librarian-puppet, a “build” step, a...
22
alias puppet-test='sudo puppet agent --test --noop --
environment'
That deploys an environment titled UserBranch, camel...
‣ Jenkins also runs a test suite in parallel
‣ Some day that’ll be a precursor to the deploy
job. Some day…
23
" Sometimes you just want to
do something to prod
‣ All that well-defined procedure means time for
quick fixes goes up
‣ Mak...
‣ Possible answers here:
• Pull some things out into their own repos
• Accept that it’s slower
• Tool it up, make the mach...
! MCollective is still awesome
26
Original image: http://xkcd.com/353/
Cheapo editing courtesy OS X Preview “Annotate”
Sam...
It’s like Frank’s Red Hot
‣ Our deploys (not IT, app teams) now use it to
figure out which machines to deploy to
‣ We wrote...
" ActiveMQ is hard
‣ MCollective sits on top of ActiveMQ
‣ Easy, right? Just install it (with Puppet, even)
and go to town...
A sad example
29
Suddenly, one day…
‣ Pretend x is time
‣ y is memory usage
‣ That’s our activeMQ
• At 2AM
• Every day
30
31
“Sudden Clarity Clarence” via memecreator.eu
Original image? Who knows. Might be Clarence.
‣ Long story short: make really sure your broker
names are unique across your network
‣ Really really sure
32
That won’t happen to us
‣ Maybe not
‣ Just know you’re adding another service you rely
on
‣ Monitor it
‣ Measure it
‣ Have...
‣ R.I.’s scripts [1]
‣ Erwan Ben Souiden, aka labynocle [2]
‣ A script I butchered to help
34
[1]: https://github.com/ripi...
! Cloudstack!
‣ It’s great
‣ Setup was easy
‣ Only using it in test now (so we only kinda care if
the machines burn down)
...
‣ Remember that puppet-test alias?
‣ That + CloudStack means people can build
machines to test Puppet changes to
‣ I was s...
You don’t have to use CloudStack,
but you need something like it
‣ That could be OpenStack, Vagrant, even some
AWS credit
...
" Would be better in prod
‣ We’re not yet running CloudStack in production.
It’ll happen someday.
‣ So, while I’m singing ...
! create_resources
‣ I have to show off one chunk of code
‣ Let me quote from TFM:
39
‣ That’s vague, right? The example i...
I already said something about Frank’s Red Hot, right?
Almost everywhere I use it looks like this
40
$thing = loadYaml(“${...
‣ One define uses the lastOctet and some
knowledge about our systems to set up an
interface on the load balancers
‣ Another...
‣ Another benefit: it’s YAML. You don’t have to
write it by hand
‣ Grab some external data source, frob as
needed, dump out...
Huge gotcha here
‣ Everything you pass that YAML to needs to do
something with all the parameters named in the file
‣ Addin...
" Maybe not everything goes
in Puppet
‣ Last year I had this slide (included verbatim)
44
Well, I can’t always be right
45
‣ Think to yourself: do I ever want to do this thing
without running Puppet?
‣ Does it make sense for this to go through t...
‣ As we’ve grown to more sites and more Puppet
masters, this has hurt all over the place
‣ The deploy concept (vs `svn up`...
Examples
‣ DNS data: there’s only one actual copy of your
DNS, and that’s what the world can see. What
does a beta machine...
</TechnicalContent>
49
Want to help out?
We’re hiring
50
‣ Sysadmins
‣ Developers (Rails or Java)
‣ Testers
‣ Lots more
backstopsolutions.com/car...
DevOpsDays Chicago!
‣ October 7th and 8th @ the Sears Tower
‣ CFP open until the 22nd
‣ Registration open now! Use code
PU...
Puppet Users Group
‣ Hosted by this guy, downtown, ~monthly
(coming soon!)
‣ meetup.com/Chicago-Puppet-users-group/
52
That’s it
Questions?
53
Upcoming SlideShare
Loading in …5
×

Puppet Camp Chicago 2014: Puppet at backstop another year of lessons

688 views

Published on

Puppet Camp Chicago 2014: "Puppet at backstop another year of lessons" by Bill Weiss, Backstop

Published in: Software
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
688
On SlideShare
0
From Embeds
0
Number of Embeds
49
Actions
Shares
0
Downloads
7
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Puppet Camp Chicago 2014: Puppet at backstop another year of lessons

  1. 1. Puppet at Backstop Another year of lessons 1
  2. 2. $ whoami ‣ Bill Weiss <bill@backstopsolutions.com> ‣ @BillWeiss ‣ Probably responsible for 90% of the badness in this talk, 25% of the good ‣ Still not a designer 2
  3. 3. What’s this talk about? aka, who cares? 3
  4. 4. 4 Image via MemeCrunch. Original owned by the History Channel.
  5. 5. No, wait, that’s not it (Come talk to me afterward about that) 5
  6. 6. ‣ I’m here to talk about things that work (and don’t) for us in our environment ‣ Hopefully most of it is useful to you ‣ Some might just be there so you can point and laugh 6
  7. 7. –You “What the heck is Backstop?” 7
  8. 8. Super helpful, I know 8
  9. 9. SaaS for institutional investment firms ‣ That’s it. ‣ You’re probably not our target audience ‣ If you are, awesome, let’s talk later 9
  10. 10. Some numbers ‣ ~195 Puppeted machines in production (US) ‣ ~350 total ‣ 2 datacenters, 2 AWS AZs, 1 office closet full of machines ‣ 50+ Engineering folks (IT, dev, QA, etc) Numbers of machines from MCollective! 10
  11. 11. Logo owned by the CentOS project Logo also owned by Oracle, of course Logo owned by Oracle Logo by David Heinemeier Hansson 11
  12. 12. ‣ Big application is J2EE on JBoss ‣ Lots of smaller bits (“services”) on Tomcat ‣ Second app is Ruby on Rails 12
  13. 13. How ‘bout that Puppet? ‣ Moved to GitHub 2013/11/18 ‣ ~4100 commits since then (~15 commits/day!) ‣ 2700 from people outside sysadmin team ‣ ~1350 pull requests from 21 contributors 13
  14. 14. The move to GitHub was awesome ‣ 4 years of Subversion before that had 29 authors and 6700 commits ‣ Other things at play here beside just s/svn/git/ , but it helped a lot ‣ How? 14
  15. 15. ! git workflow ‣ Org-owned repo, two branches ‣ master ➡ production ‣ office ➡ testing (pre-prod) ‣ Everyone forks from master and sends pull requests to office ‣ Pull changes from office to master as needed 15
  16. 16. ‣ Everyone has their own fork with work branches ‣ Branch names probably are Jira ticket numbers ‣ Everyone bases from master, not office, so we can pull changes independently 16
  17. 17. This is why we have so many contributors 17
  18. 18. ‣ If you’re a software shop, you already have a workflow. Do that ‣ Otherwise, this is working well for us 18
  19. 19. " git hooks ‣ We had a pretty good set of pre-commit hooks in svn to test changes ‣ Github doesn’t do that # ‣ Git pre-commit hooks exist, but it’s up to each person to set them up ‣ No awesome answer yet ‣ The answer might be… 19
  20. 20. ! Jenkins! ‣ We used to have this “deploy” job on the Puppet masters as a cron job: • (cd /etc/puppet && svn up) ‣ Nothing could go wrong there, right? ‣ For a while we just did it manually instead 20
  21. 21. ‣ Jenkins polls for changes to either branch ‣ Runs a deploy job that grabs repo, runs librarian-puppet, a “build” step, and then pushes it out to the masters ‣ Determines masters via mco call ‣ All the good (and bad) of capistrano for the push 21
  22. 22. 22 alias puppet-test='sudo puppet agent --test --noop -- environment' That deploys an environment titled UserBranch, camel-cased
  23. 23. ‣ Jenkins also runs a test suite in parallel ‣ Some day that’ll be a precursor to the deploy job. Some day… 23
  24. 24. " Sometimes you just want to do something to prod ‣ All that well-defined procedure means time for quick fixes goes up ‣ Make a branch, make some changes, commit, push, PR to office, merge, PR to prod, merge, run deploy… 24
  25. 25. ‣ Possible answers here: • Pull some things out into their own repos • Accept that it’s slower • Tool it up, make the machines do the slow parts 25
  26. 26. ! MCollective is still awesome 26 Original image: http://xkcd.com/353/ Cheapo editing courtesy OS X Preview “Annotate” Same graphic as last year’s talk. I love you, mco!
  27. 27. It’s like Frank’s Red Hot ‣ Our deploys (not IT, app teams) now use it to figure out which machines to deploy to ‣ We wrote an agent for our VM system (Proxmox) Which, uh, evidently we haven’t released. Will do. ‣ Stop/start the world for Oracle maintenance 27
  28. 28. " ActiveMQ is hard ‣ MCollective sits on top of ActiveMQ ‣ Easy, right? Just install it (with Puppet, even) and go to town ‣ Second datacenter? Yeah, just get them talking to each other and… 28
  29. 29. A sad example 29
  30. 30. Suddenly, one day… ‣ Pretend x is time ‣ y is memory usage ‣ That’s our activeMQ • At 2AM • Every day 30
  31. 31. 31 “Sudden Clarity Clarence” via memecreator.eu Original image? Who knows. Might be Clarence.
  32. 32. ‣ Long story short: make really sure your broker names are unique across your network ‣ Really really sure 32
  33. 33. That won’t happen to us ‣ Maybe not ‣ Just know you’re adding another service you rely on ‣ Monitor it ‣ Measure it ‣ Have a game plan for what to do when it goes horribly wrong 33
  34. 34. ‣ R.I.’s scripts [1] ‣ Erwan Ben Souiden, aka labynocle [2] ‣ A script I butchered to help 34 [1]: https://github.com/ripienaar/monitoring-scripts/tree/master/activemq [2]: https://github.com/labynocle/nagios_plugins_by_labynocle/tree/master/check_activeMQ [3]: https://github.com/BillWeiss/MiscScripts/blob/master/check_activemq.pl I know, I know, bleh, Nagios
  35. 35. ! Cloudstack! ‣ It’s great ‣ Setup was easy ‣ Only using it in test now (so we only kinda care if the machines burn down) 35 That’s the CloudStack monkey, owned by the Apache Software Foundation
  36. 36. ‣ Remember that puppet-test alias? ‣ That + CloudStack means people can build machines to test Puppet changes to ‣ I was surprised recently by a service that just showed up ready to run in prod, Puppet and all 36
  37. 37. You don’t have to use CloudStack, but you need something like it ‣ That could be OpenStack, Vagrant, even some AWS credit ‣ You just need fast to build disposable machines 37
  38. 38. " Would be better in prod ‣ We’re not yet running CloudStack in production. It’ll happen someday. ‣ So, while I’m singing its praises, I can’t say we’re all in. 38
  39. 39. ! create_resources ‣ I have to show off one chunk of code ‣ Let me quote from TFM: 39 ‣ That’s vague, right? The example in TFM didn’t inspire me either, so I glossed over it for years. Converts a hash into a set of resources and adds them to the catalog.
  40. 40. I already said something about Frank’s Red Hot, right? Almost everywhere I use it looks like this 40 $thing = loadYaml(“${yamlDir}/thing/${environment}.yaml”)
 create_resources( some::define, $thing ) That same YAML gets used in a bunch of those. backstopsolutions:
 lastOctet: 226
 clientFqdn: 'www.backstopsolutions.com'
 seconddn: 'backstopsolutions.com'
 includeFile: 'true'
  41. 41. ‣ One define uses the lastOctet and some knowledge about our systems to set up an interface on the load balancers ‣ Another uses the hostnames to set up a couple of vhosts for the Rails app ‣ Yet another adds it into several DNS zones 41
  42. 42. ‣ Another benefit: it’s YAML. You don’t have to write it by hand ‣ Grab some external data source, frob as needed, dump out some YAML. As long as you meet the spec, you’ll be fine 42
  43. 43. Huge gotcha here ‣ Everything you pass that YAML to needs to do something with all the parameters named in the file ‣ Adding a field thus means changing all those defines ‣ We have a lot of ‘$foo = “dontcare”’ in there. Just come up with an idiom and stick to it ‣ You could probably do something clever to filter out params? 43
  44. 44. " Maybe not everything goes in Puppet ‣ Last year I had this slide (included verbatim) 44
  45. 45. Well, I can’t always be right 45
  46. 46. ‣ Think to yourself: do I ever want to do this thing without running Puppet? ‣ Does it make sense for this to go through the same test -> prod cycle? ‣ Do the same people need to see this content as the Puppet manifests? ‣ Can it be generated in one place, or do you need to gather data from several? 46
  47. 47. ‣ As we’ve grown to more sites and more Puppet masters, this has hurt all over the place ‣ The deploy concept (vs `svn up`) made a mess of the checkin of this derived data ‣ I still think it’s an OK approach, but it needs thought 47
  48. 48. Examples ‣ DNS data: there’s only one actual copy of your DNS, and that’s what the world can see. What does a beta machine do with it? ‣ Load balancer configs: multiple datacenters need to know about each other, but can’t see the other’s database to derive config from 48
  49. 49. </TechnicalContent> 49
  50. 50. Want to help out? We’re hiring 50 ‣ Sysadmins ‣ Developers (Rails or Java) ‣ Testers ‣ Lots more backstopsolutions.com/careers
  51. 51. DevOpsDays Chicago! ‣ October 7th and 8th @ the Sears Tower ‣ CFP open until the 22nd ‣ Registration open now! Use code PUPPET_CAMP for 10% off
 devopsdays.org/events/2014-chicago/registration/ ‣ I might be on the committee 51 I refuse to call it the Willis Tower, y’all
  52. 52. Puppet Users Group ‣ Hosted by this guy, downtown, ~monthly (coming soon!) ‣ meetup.com/Chicago-Puppet-users-group/ 52
  53. 53. That’s it Questions? 53

×