Jesse Newland             jnewlandhey errbodymy name is jesse newlandI do ops at GitHub
Puppet                                            at                    GitHubAnd today I’m going to be talking about Pupp...
All of the amazing Puppet  OSS projects @rodjek has written but doesn’t    want to talk aboutFirst... I’ll be talking abou...
*And then, I want to introduce you to the star of the GitHub Ops team, Hubot, and tell you alittle bit about something we’...
the            SetupBut, before I get into all of that, Im actually going to talk about anupcoming talk, one by a coworker...
Puppet                                          at                   GitHubSo I guess you could say that I’m talking about
THE of          REST                     Puppet                                         at                   GitHubthe res...
4 years, >100k LOCWe’ve been managing GitHub’s infrastructure with Puppet for 4 years, since the move toRackspace. There’s...
SimpleBut we are obsessed with keeping our Puppet deployment simple
Single             MasterWe use a single puppetmaster running lots of unicorns. Nothing fancy. It works for now.However, w...
cron FTW # cat /etc/cron.d/puppet 13 * * * * root /usr/bin/We don’t run the agent, but rather run puppet on cron every hou...
No                            ENCWe don’t use an external node classifier
([a-z0-9-_]+)(d+)([a-z]?).(.*).github.com $ cat manifests/nodes/janky.rscloud.pp node /^jankyd+.rscloud.github.com$/ {   g...
Where the magic happens$ head modules/github/manifests/role/janky.ppdefine github::role::janky($public_address,           ...
Heavy use of augeas  augeas { my.cnf/avoid_cardinality_skew:    context => /files/etc/mysql/my.cnf/mysqld/,    changes => ...
BORINGBut I don’t want to just show all of you Puppet code for thirty minutes. Thats boring
What’s interesting   about Puppet at      GitHub?I’d rather talk about whats interesting about how we use Puppet at GitHub...
Making     Puppet Less        ScaryWe’re doing our best to make puppet less scary for people that aren’t familiar with it,...
I’ve been thinking about this a lot recently as we’ve just had two large infrastructure projectsshipped by people that wer...
And Adam Roben shipped puppet manifests for our windows build and CI servers.
this                               is                              goodThis is an awesome trend, and I want it to continue...
Flow just like  a (GitHub)Ruby projectFor us, an important part of making Puppet development accessible for other develope...
Setup  $ ./script/bootstrapLike making it as easy to setup as any other project at GitHub
$ cat Gemfile        source :rubygems        gem     puppet,             2.7.18        gem     facter,             1.6.10 ...
$ cat Puppetfileforge "http://forge.puppetlabs.com"mod puppetlabs/apt...And puppet deps are managed by librarian-puppet, a...
          rodjek / librarian-puppetFor those of you keeping score at home, that’s the first of Tim Sharpe’s open source pr...
Making puppet flow like other projects at GitHub means ensuring we have good editorsupport for the language
 rodjek / vim-puppetvim-puppet, that’s two.
Tests  $ ./script/cibuildIt means running tests is a simple one-step process
TESTS!Tests are super important. A solid and easy to use test harness helps build developerconfidence in a new language.
Safety             netAnd tests are crucial safety net for helping people cut their teeth on Puppet if they haven’tever to...
rspec-puppet  should contain_github__firewall_rule(internal_network)  should contain_ssmtp__relay_to(smtp).with_relay_host...
 rodjek / rspec-puppetrspec-puppet, that’s three
role describe github::role::fe do   let(:title) { fe }   let(:node) { fe1.rs.github.com }   let(:params) {     {          ...
.git/hooks/pre-commit $ git commit -am "lolbadchange" modules/github/manifests/role/fe.pp:err: Could not parse for environ...
 rodjek / puppet-lintpuppet-lint, that’s four, btw.
specs run on each push          auto deploy on CI passrspec-puppet and puppet-lint are automatically run by CI on every co...
As you can see, Hubot automates a lot of the process of rolling out PuppetThat example covered pushing changes to master, ...
Say we have a pull request for a branch we want to merge, and that we’ve reviewed the codeand it all looks good.
branches                                  ==   environmentsOn each deploy, we turn all git branches into puppet environmen...
This combined with heaven, our capistrano-powered deployment API we interact with viaHubot, enables us to experiment with ...
So, to safely merge this pull request...
hubot ci status puppet/git-gh13            deploy:apply puppet/git-gh13 staging/fs1            deploy:noop puppet/git-gh13...
Build #108816   (5fe75932f26ea62cb5fc5e3d0cb302cc2461d11e)   of puppet/git-gh13 was successful(421s) github/              ...
hubot ci status puppet/git-gh13             deploy:apply puppet/git-gh13 staging/fs1             deploy:noop puppet/git-gh...
**  [out :: REDACTED ] Bootstrapping...  **  [out :: REDACTED ] Gem environment up-to-date.  **  [out :: REDACTED ] Runnin...
hubot ci status puppet/git-gh13             deploy:apply puppet/git-gh13 staging/fs1             deploy:noop puppet/git-gh...
**  [out :: REDACTED ] Bootstrapping...  **  [out :: REDACTED ] Gem environment up-to-date.  **  [out :: REDACTED ] Runnin...
hubot ci status puppet/git-gh13             deploy:apply puppet/git-gh13 staging/fs1             deploy:noop puppet/git-gh...
hubot ci status puppet/git-gh13             deploy:apply puppet/git-gh13 staging/fs1             deploy:noop puppet/git-gh...
**  [out :: REDACTED ] Bootstrapping...  **  [out :: REDACTED ] Gem environment up-to-date.  **  [out :: REDACTED ] Runnin...
hubot ci status puppet/git-gh13            deploy:apply puppet/git-gh13 staging/fs1            deploy:noop puppet/git-gh13...
Yup, looks good
hubot ci status puppet/git-gh13            deploy:apply puppet/git-gh13 staging/fs1            deploy:noop puppet/git-gh13...
Yup, looks good
ChatOpsHow we interact with Puppet via Hubot is a great example of a core principal of how we doops at GitHub. We’ve been ...
Essentially, ChatOps is the result of Hubot becoming sentient, and decreeing, among otherthings, that we now address him a...
                        HubotActually, that’s not it at all. Hubot is the star of our Ops team.
heaven       shell               Hubot                                       janky                                graphmeW...
ALL OF  hubotshell                                   heaven                                                               ...
Why is this stupid          chat bot so       important to Ops?But why do we obsess about Hubot so much? It’s just a chat ...
hubot ci status puppet/git-gh13             deploy:apply puppet/git-gh13 staging/fs1             deploy:noop puppet/git-gh...
Everyone sees all    of that happen   on their first dayEveryone sees all of this happen from the minute they join GitHub....
You don’t just see how to roll out puppet, you see how to...
hubot ci status github/smoke-perfcheck the status of branch’s last build
hubot deploy github/smoke-perf to prod/fe1deploy a any branch of any github app to any server
hubot graph me -10min @app-perfget graphs of the app’s recent performance
hubot procs unicorncheck the status of unicorns across all frontends
hubot resque criticalcheck the status of the resque critical queue
hubot graph me -10min @collectd.load(fe*)check load on the frontends
hubot conns fe1check current connections to a frontend that you suspect has a problem
hubot log me smoke fe1grab smoke logs for that frontend and realize that you did, in fact, break it
hubot lbctl disable fe1take it out of the load balancer
hubot status yellow Bad deploy. Reverting now.update the status blog
hubot who’s on calldetermine who is currently on call so you can apologize to them
hubot pingdom checkscheck pingdom to make sure you haven’t broken everything
hubot upset mechill yourself out really quick
hubot deploy github to prod/fe1revert back to master on the busted frontend
hubot log me smoke fe1verify things have returned to normal
hubot air drum meget pumped up because you fixed it
hubot lbctl enable fe1bring the fixed frontend back into the rotation
hubot status green All systems go.clear alerts on the status page
hubot whois 4.9.23.22Once the outage has been resolved, you might see how to grab whois information for an IPthat exhibite...
hubot khanify spammersand how to hit meme generator to make a joke when you realize that IP is a spammer
hubot play in the air tonightthen someone would queue up the song that popped into their head when they thoughtabout drums...
hubot tweet@github PuppetConf Drinkup Friday                    night at 8:30 at Zeke’s                    (3rd & Brannan)...
ChatOpsChatOps means building tools that make it easier to operate your infrastructure via Hubotthan via Terminal or Chrome
By placing tools              directly in the              middle of the              conversationBecause...
Everyone              is pairing           all of the timeThis is the core concept behind ChatOps.
Teaching                      by   doingTeaching by doing is awesome
This was always my mainmotivation with hubot - teaching  by doing by making things    visible. Its an extremely       powe...
This is how I respond to “how to I do X” questions in Campfire now.If there’s not yet Hubot functionality to do a thing, we...
Communicate    by                  doingPlacing tools in the middle of the conversation also means you get communication o...
THINGS IHAVEN’T ASKED  RECENTLYFor example, here are a few things I haven’t asked recently because Hubot has told me thean...
THINGS IHAVEN’T ASKED how’s that deploy going?    RECENTLY
THINGS I  are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?    RECENTLY
THINGS I  are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone responding to that nagios ale...
THINGS I   is that branch green?  are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone respo...
THINGS I   is that branch green?  are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone respo...
did anyone update the status page?     THINGS I   is that branch green?  are you deploying that or should i?HAVEN’T ASKED ...
did anyone update the status page?     THINGS I   is that branch green?  are you deploying that or should i?HAVEN’T ASKED ...
Free communication is especially crucial in a distributed environment.
Our Ops team is entirely remote, so Campfire is our default means of communication.
http://www.flickr.com/photos/7997249@N06/6061305639/This is extremely helpful during outages or other situations that requ...
Hide            the                        uglyAnother awesome benefit of ChatOps-ing all of the things is that you can hid...
My favorite example of this is ugliest of the ugly, Nagios.
[nines] hubot opened issue #4263: Nagios   (229906) - fs3b/syslog - Tue Sept 25 23:40:18   PDT 2012. github/nines#4263Hubo...
hubot nagios ack fs3b/syslog  # fix stuff               nagios check fs3b/syslog               nagios status fs3b/syslog  ...
Mobile              FTWYet another awesome benefit of ChatOps is that you get mobile support for free
Well, that is, if you have a team of awesome iOS developers that have built an actuallyfunctioning Campfire client for the ...
ChatOpsThat’s ChatOps at its finest.
And now for                  something                  completely                   differentWhile I’m showing off mobile...
We’ve hacked together support for PagerDuty alerts via Apple Push Notifications. When youswipe on the alert, you go directl...
Which lets you ack an alert
while you’re still in bed
or on the couch.
BoomI can’t even begin to tell you how happy this makes me, and how less shitty it makes beingon-call
So, who better to summarize all of this than Hubot himself. I asked him what he thoughtabout ChatOps. Here’s what he said:
ChatOps all the things.Listen to what Hubot said. You’ll love it. Your ops team will love it.And you’ll help other develop...
Work at GitHub      jesse@github.comIf you can’t ChatOps all the things at your gig now, you could always just come work w...
Thanks!That’s all I have. Thanks for listening! any questions?
Tomorrow @ 8:30 PM                                                               Zeke’s                     3rd & BrannanW...
Puppet at GitHub / ChatOps
Upcoming SlideShare
Loading in...5
×

Puppet at GitHub / ChatOps

42,694

Published on

"Puppet at GitHub / ChatOps" from PuppetConf 2012, by Jesse Newland

Video of "Puppet at GitHub": http://bit.ly/WVS3vQ
Learn more about Puppet: http://bit.ly/QQoAP1

Abstract: Ops at GitHub has a unique challenge - keeping up with the rabid pace of features and products that the GitHub team develops. In this talk, we'll focus on tools and techniques we use to rapidly and confidently ship infrastructure changes/features with Puppet using Puppet-Rspec, CI, Puppet-Lint, branch puppet deploys, and Hubot.

Speaker Bio: Jesse Newland does Ops at GitHub. His favorite hobby is SPOF wack-a-mole, followed closely by guitar and piano. Prior to GitHub, Jesse was the CTO at Rails Machine where he ran a large private cloud and managed several hundred production Ruby on Rails applications using Puppet. To the delight and/or chagrin of the Puppet community, Jesse is to blame for Moonshine, the Ruby DSL for Puppet before Puppet had a Ruby DSL.

Published in: Technology
2 Comments
10 Likes
Statistics
Notes
No Downloads
Views
Total Views
42,694
On Slideshare
0
From Embeds
0
Number of Embeds
30
Actions
Shares
0
Downloads
181
Comments
2
Likes
10
Embeds 0
No embeds

No notes for slide

Transcript of "Puppet at GitHub / ChatOps"

  1. 1. Jesse Newland jnewlandhey errbodymy name is jesse newlandI do ops at GitHub
  2. 2. Puppet at GitHubAnd today I’m going to be talking about Puppet at GitHub.Really, I’m telling a story in two parts.
  3. 3. All of the amazing Puppet OSS projects @rodjek has written but doesn’t want to talk aboutFirst... I’ll be talking about all of the amazing Puppet open source projects Tim Sharpe haswritten but doesn’t want to talk aboutand how we use them at GitHub
  4. 4. *And then, I want to introduce you to the star of the GitHub Ops team, Hubot, and tell you alittle bit about something we’ve been calling ChatOps
  5. 5. the SetupBut, before I get into all of that, Im actually going to talk about anupcoming talk, one by a coworker of mine at GitHub. Will Farringtonis going to be speaking tomorrow at 2:45pm about The Setup, ourPuppet-powered GitHubber laptop management solution. Itsamazing. Its one of the coolest uses of Puppet Ive ever seen, andits going to completely change the way you think about yourdevelopment environment.But I’m not going to be talking about any of that today.So, yeah, go to Wills talk tommorrow. You wont be disappointed.
  6. 6. Puppet at GitHubSo I guess you could say that I’m talking about
  7. 7. THE of REST Puppet at GitHubthe rest of puppet at github. For the scope of this talk, I’m going to be talking about thePuppet infrastructure that runs github.com
  8. 8. 4 years, >100k LOCWe’ve been managing GitHub’s infrastructure with Puppet for 4 years, since the move toRackspace. There’s a ton of code, and we’re developing at a rapid pace.
  9. 9. SimpleBut we are obsessed with keeping our Puppet deployment simple
  10. 10. Single MasterWe use a single puppetmaster running lots of unicorns. Nothing fancy. It works for now.However, we will need to scale this tier up or out in about 6 months if the trends look right.We’ll probably switch to two load balanced puppetmasters around that time.
  11. 11. cron FTW # cat /etc/cron.d/puppet 13 * * * * root /usr/bin/We don’t run the agent, but rather run puppet on cron every hour in combination with runstriggered via Hubot (more on that later)
  12. 12. No ENCWe don’t use an external node classifier
  13. 13. ([a-z0-9-_]+)(d+)([a-z]?).(.*).github.com $ cat manifests/nodes/janky.rscloud.pp node /^jankyd+.rscloud.github.com$/ { github::role::janky { janky: public_address => dns_lookup($fqdn), nginx_hostname => $fqdn, } }Instead, we give nodes DNS names that adhere to a naming convention that maps them to apre-defined role
  14. 14. Where the magic happens$ head modules/github/manifests/role/janky.ppdefine github::role::janky($public_address, $nginx_hostname=, $god=true ) { github::core { janky: } include github::app::janky github::nginx { janky: }}Role definitions are where the magic happens. We try to DRY common functionality into ourcore module and into other simple classes or defines so that role definitions read like a nicesummary of what makes this role different from others
  15. 15. Heavy use of augeas augeas { my.cnf/avoid_cardinality_skew: context => /files/etc/mysql/my.cnf/mysqld/, changes => [ set innodb_stats_auto_update 0, set innodb_stats_on_metadata 0, set innodb_stats_on_metadata 64 ], require => Percona::Server[$::fqdn], }We generally try to avoid templates for configuration files in favor of using aw ge usLets us manage the small pieces of configuration we care about and use the OS defaults forthe things we dont.
  16. 16. BORINGBut I don’t want to just show all of you Puppet code for thirty minutes. Thats boring
  17. 17. What’s interesting about Puppet at GitHub?I’d rather talk about whats interesting about how we use Puppet at GitHub. And what I thinkis the most interesting is that we focus heavily on ensuring the Puppet development workflowis easily accessible to everyone at GitHub.
  18. 18. Making Puppet Less ScaryWe’re doing our best to make puppet less scary for people that aren’t familiar with it, so theycan help the Ops team grow and evolve our infrastructure. We’re doing some things righthere, but there’s still a lot of work to do.
  19. 19. I’ve been thinking about this a lot recently as we’ve just had two large infrastructure projectsshipped by people that were completely or relatively new to puppet. First, Derek Greentreeshipped a Cassandra cluster,,,
  20. 20. And Adam Roben shipped puppet manifests for our windows build and CI servers.
  21. 21. this is goodThis is an awesome trend, and I want it to continue. So I thought I’d talk a bit today aboutwhat we’re doing to try to enable even more of this.
  22. 22. Flow just like a (GitHub)Ruby projectFor us, an important part of making Puppet development accessible for other developers atGitHub is making the development flow on our puppet codebase as similar as possible to thatof any other GitHub Ruby project. That means sticking with some common conventions
  23. 23. Setup $ ./script/bootstrapLike making it as easy to setup as any other project at GitHub
  24. 24. $ cat Gemfile source :rubygems gem puppet, 2.7.18 gem facter, 1.6.10 gem rspec-puppet, 0.1.2 gem rake, 0.8.7 gem puppet-lint, 0.2.1 gem ruby-augeas, 0.3.0 gem json, 1.5.1 gem fog, 1.3.1 gem librarian-puppet, 0.9.4 gem parallel_testsSo ruby deps are managed by Bundler
  25. 25. $ cat Puppetfileforge "http://forge.puppetlabs.com"mod puppetlabs/apt...And puppet deps are managed by librarian-puppet, a bundler-like library that manages thepuppet modules your infrastructure depends on and install them directly from GitHubrepositories.I’m of the opinion that the unit of open source currency is no longer a tarball downloadedfrom a something named *forge. It’s a GitHub repo. All of the developers at GitHub feel thesame way, so Tim wrote librarian puppet
  26. 26.  rodjek / librarian-puppetFor those of you keeping score at home, that’s the first of Tim Sharpe’s open source projectsthat I’ve mentioned. Hi Tim!
  27. 27. Making puppet flow like other projects at GitHub means ensuring we have good editorsupport for the language
  28. 28.  rodjek / vim-puppetvim-puppet, that’s two.
  29. 29. Tests $ ./script/cibuildIt means running tests is a simple one-step process
  30. 30. TESTS!Tests are super important. A solid and easy to use test harness helps build developerconfidence in a new language.
  31. 31. Safety netAnd tests are crucial safety net for helping people cut their teeth on Puppet if they haven’tever touched it before.
  32. 32. rspec-puppet should contain_github__firewall_rule(internal_network) should contain_ssmtp__relay_to(smtp).with_relay_host(smtp) should contain_file(/etc/logstash/logstash.conf) should include_class(github::ksplice) should contain_networking__bond(bond0).with( :gateway => 172.22.0.2, :arp_ip_target => 172.22.0.2, :up_commands => nil )We use rspec-puppet heavily. If you haven’t used rspec-puppet yet, go check it out rightnow.It’s amazing.There are no less than three talks about it at Puppetconf, so I’m not going to talk about HOWto use it today, just touch a little bit on how WE use it.
  33. 33.  rodjek / rspec-puppetrspec-puppet, that’s three
  34. 34. role describe github::role::fe do let(:title) { fe } let(:node) { fe1.rs.github.com } let(:params) { { specs :public_address => 207.97.227.242/27, :private_address => 172.22.1.59/22, :git_weight => 16 } } let(:facts) { are { :ipaddress => 172.22.1.59, :operatingsystem => Debian, :datacenter => rackspace-iad2, } king } it do should contain_github__core(fe) ... end endWe try our best to adequately test our individual puppet modules, but our central and mostfrequently touched specs exercise our role system. There’s one spec for each role whichdescribes its intended functionality.These specs focus on critical functionality of each role, and help a great deal to buildconfidence that we’re not introducing regressions when adding or refactoring functionality orworking in other roles.
  35. 35. .git/hooks/pre-commit $ git commit -am "lolbadchange" modules/github/manifests/role/fe.pp:err: Could not parse for environment production: Syntax error at allow_outbound_syslog; expected } at /Users/jnewland/github/puppet/modules/github/ manifests/role/fe.pp:31 modules/github/manifests/role/fe.pp - WARNING: => is not properly aligned on line 626For an even faster feedback loop than running specs, all Puppet dev environmentsautomatically get setup with a pre-commit hook that checks for syntax errors and ensuresyour changes confirm to the Puppet Style guide.This has proved amazingly useful for Puppet novices and experts alike, novices finding ithelps them understand language conventions quickly and guides them towards solutions,and experts using it to catch typos and help them not look like novices.
  36. 36.  rodjek / puppet-lintpuppet-lint, that’s four, btw.
  37. 37. specs run on each push auto deploy on CI passrspec-puppet and puppet-lint are automatically run by CI on every commit on every branchpushed to our Puppet repo.Once master passes CI, puppet is automatically deployed
  38. 38. As you can see, Hubot automates a lot of the process of rolling out PuppetThat example covered pushing changes to master, but what about a Pull-Request basedworkflow?
  39. 39. Say we have a pull request for a branch we want to merge, and that we’ve reviewed the codeand it all looks good.
  40. 40. branches == environmentsOn each deploy, we turn all git branches into puppet environments.
  41. 41. This combined with heaven, our capistrano-powered deployment API we interact with viaHubot, enables us to experiment with unmerged Puppet branches in a powerful way
  42. 42. So, to safely merge this pull request...
  43. 43. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/githubYou might ask Hubot to confirm its build status
  44. 44. Build #108816 (5fe75932f26ea62cb5fc5e3d0cb302cc2461d11e) of puppet/git-gh13 was successful(421s) github/ puppet@567ea48...5fe7593Yup, looks good.
  45. 45. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/githubThen roll the branch out to a staging box to make everything applies cleanly there.
  46. 46. ** [out :: REDACTED ] Bootstrapping... ** [out :: REDACTED ] Gem environment up-to-date. ** [out :: REDACTED ] Running librarian-puppet... ** [out :: REDACTED ] Generating puppet environments... ** [out :: REDACTED ] Cleaning up deleted branches... ** [out :: REDACTED ] Done! ** [out :: REDACTED ] Sending restart command ** [out :: REDACTED ] The following watches were affected: ** [out :: REDACTED ] puppetmaster_unicorn ** [out :: fs1a.stg.github.com] info: Applying configuration version 8fb1a2716d5f950b836e511471a2bdac3ed27090 ** [out :: fs1a.stg.github.com] notice: /Stage[main] Github::Common_packages/Package[git]/ensure: ensure changed 1:1.7.10-1+github12 to 1:1.7.10-1+github13 ...Yup, looks good.
  47. 47. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/githubThen, if you wanted an extra layer of confidence, you could noop the branch against aproduction node
  48. 48. ** [out :: REDACTED ] Bootstrapping... ** [out :: REDACTED ] Gem environment up-to-date. ** [out :: REDACTED ] Running librarian-puppet... ** [out :: REDACTED ] Generating puppet environments... ** [out :: REDACTED ] Cleaning up deleted branches... ** [out :: REDACTED ] Done! ** [out :: REDACTED ] Sending restart command ** [out :: REDACTED ] The following watches were affected: ** [out :: REDACTED ] puppetmaster_unicorn ** [out :: fs1a.rs.github.com] info: Applying configuration version 8fb1a2716d5f950b836e511471a2bdac3ed27090 ** [out :: fs1a.rs.github.com] notice: /Stage[main]/ Github::Common_packages/Package[git]/ensure: would have changed from 1:1.7.10-1+github12 to 1:1.7.10-1+github13 ...Yup, looks good
  49. 49. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/githubNext, you’d merge the pull request. If you stopped here, the code would gradually roll out toall affected nodes over the next hour.
  50. 50. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/githubIf you wanted the rollout to happen faster than that, you could force a puppet run on theaffected class of nodes
  51. 51. ** [out :: REDACTED ] Bootstrapping... ** [out :: REDACTED ] Gem environment up-to-date. ** [out :: REDACTED ] Running librarian-puppet... ** [out :: REDACTED ] Generating puppet environments... ** [out :: REDACTED ] Cleaning up deleted branches... ** [out :: REDACTED ] Done! ** [out :: REDACTED ] Sending restart command ** [out :: REDACTED ] The following watches were affected: ** [out :: REDACTED ] puppetmaster_unicorn ** [out :: fs1a.rs.github.com] info: Applying configuration version 8fb1a2716d5f950b836e511471a2bdac3ed27090 ** [out :: fs7b.rs.github.com] info: Applying configuration version 8fb1a2716d5f950b836e511471a2bdac3ed27090 ** [out :: fs1a.rs.github.com] notice: /Stage[main]/ Github::Common_packages/Package[git]/ensure: ensure changed 1:1.7.10-1+github12 to 1:1.7.10-1+github13 ** [out :: fs7b.rs.github.com] notice: /Stage[main]/ Github::Common_packages/Package[git]/ensure: ensure changed 1:1.7.10-1+github12 to 1:1.7.10-1+github13 ...Yup, that looks good.
  52. 52. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/githubThen you’d probably want to check out load to make sure nothing went crazy
  53. 53. Yup, looks good
  54. 54. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/github...and maybe check some logs or other related metrics to confirm your change didn’t breaksomething
  55. 55. Yup, looks good
  56. 56. ChatOpsHow we interact with Puppet via Hubot is a great example of a core principal of how we doops at GitHub. We’ve been calling it ChatOps recently.
  57. 57. Essentially, ChatOps is the result of Hubot becoming sentient, and decreeing, among otherthings, that we now address him as “Supreme Leader” and communicate with ourinfrastructure though his secure channels alone.We occasionally observe him speaking in tongues that sound eerily like YouTube comments.
  58. 58.  HubotActually, that’s not it at all. Hubot is the star of our Ops team.
  59. 59. heaven shell Hubot janky graphmeWe use hubot day in day out to interact with other simple tools we’ve written over JSON apis.
  60. 60. ALL OF hubotshell heaven jankyTHE APIS graphmeHubot interacts nicely with tons of external APIs too. If you have a JSON API, making yourservice work with Hubot is a piece of cake.
  61. 61. Why is this stupid chat bot so important to Ops?But why do we obsess about Hubot so much? It’s just a chat bot, right?There are some distinct upsides to this approach we’ve notices as our use of Hubot in Opshas grown
  62. 62. hubot ci status puppet/git-gh13 deploy:apply puppet/git-gh13 staging/fs1 deploy:noop puppet/git-gh13 prod/fs1 # merge pull request hubot deploy:apply puppet to prod/fs graph me -1h @collectd.load(fs*) log me hooks github/githubRemember the flow I just showed you for rolling out puppet changes to our infrastructure?
  63. 63. Everyone sees all of that happen on their first dayEveryone sees all of this happen from the minute they join GitHub. It’s right there, in the Opsroom, right in the middle of the conversation in campfire.
  64. 64. You don’t just see how to roll out puppet, you see how to...
  65. 65. hubot ci status github/smoke-perfcheck the status of branch’s last build
  66. 66. hubot deploy github/smoke-perf to prod/fe1deploy a any branch of any github app to any server
  67. 67. hubot graph me -10min @app-perfget graphs of the app’s recent performance
  68. 68. hubot procs unicorncheck the status of unicorns across all frontends
  69. 69. hubot resque criticalcheck the status of the resque critical queue
  70. 70. hubot graph me -10min @collectd.load(fe*)check load on the frontends
  71. 71. hubot conns fe1check current connections to a frontend that you suspect has a problem
  72. 72. hubot log me smoke fe1grab smoke logs for that frontend and realize that you did, in fact, break it
  73. 73. hubot lbctl disable fe1take it out of the load balancer
  74. 74. hubot status yellow Bad deploy. Reverting now.update the status blog
  75. 75. hubot who’s on calldetermine who is currently on call so you can apologize to them
  76. 76. hubot pingdom checkscheck pingdom to make sure you haven’t broken everything
  77. 77. hubot upset mechill yourself out really quick
  78. 78. hubot deploy github to prod/fe1revert back to master on the busted frontend
  79. 79. hubot log me smoke fe1verify things have returned to normal
  80. 80. hubot air drum meget pumped up because you fixed it
  81. 81. hubot lbctl enable fe1bring the fixed frontend back into the rotation
  82. 82. hubot status green All systems go.clear alerts on the status page
  83. 83. hubot whois 4.9.23.22Once the outage has been resolved, you might see how to grab whois information for an IPthat exhibited suspicious activity in the logs you saw
  84. 84. hubot khanify spammersand how to hit meme generator to make a joke when you realize that IP is a spammer
  85. 85. hubot play in the air tonightthen someone would queue up the song that popped into their head when they thoughtabout drums and gorillas at the same time
  86. 86. hubot tweet@github PuppetConf Drinkup Friday night at 8:30 at Zeke’s (3rd & Brannan)and then finish it all off with a tweet about the Drinkup we’re throwing friday night
  87. 87. ChatOpsChatOps means building tools that make it easier to operate your infrastructure via Hubotthan via Terminal or Chrome
  88. 88. By placing tools directly in the middle of the conversationBecause...
  89. 89. Everyone is pairing all of the timeThis is the core concept behind ChatOps.
  90. 90. Teaching by doingTeaching by doing is awesome
  91. 91. This was always my mainmotivation with hubot - teaching by doing by making things visible. Its an extremely powerful teaching technique - @rtomaykoRyan Tomayko had this in mind from the very first commits to hubot, which just presented asimple wrapper around a repository of shell scripts we use for management and monitoringour infrastructure.
  92. 92. This is how I respond to “how to I do X” questions in Campfire now.If there’s not yet Hubot functionality to do a thing, we try to write it.
  93. 93. Communicate by doingPlacing tools in the middle of the conversation also means you get communication of yourwork for free.If you’re doing something in a shell or on a website, you have to do it, then tell people aboutit. If you do it with hubot, that comes free.
  94. 94. THINGS IHAVEN’T ASKED RECENTLYFor example, here are a few things I haven’t asked recently because Hubot has told me theanswer
  95. 95. THINGS IHAVEN’T ASKED how’s that deploy going? RECENTLY
  96. 96. THINGS I are you deploying that or should i?HAVEN’T ASKED how’s that deploy going? RECENTLY
  97. 97. THINGS I are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone responding to that nagios alert? RECENTLY
  98. 98. THINGS I is that branch green? are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone responding to that nagios alert? RECENTLY
  99. 99. THINGS I is that branch green? are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone responding to that nagios alert? RECENTLY how does load look?
  100. 100. did anyone update the status page? THINGS I is that branch green? are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone responding to that nagios alert? RECENTLY how does load look?
  101. 101. did anyone update the status page? THINGS I is that branch green? are you deploying that or should i?HAVEN’T ASKED how’s that deploy going?is anyone responding to that nagios alert? RECENTLY how does load look? did that deploy finish?
  102. 102. Free communication is especially crucial in a distributed environment.
  103. 103. Our Ops team is entirely remote, so Campfire is our default means of communication.
  104. 104. http://www.flickr.com/photos/7997249@N06/6061305639/This is extremely helpful during outages or other situations that require tactical response.You don’t have to SAY that you’re spraying water on the fire, people SEE you doing it.
  105. 105. Hide the uglyAnother awesome benefit of ChatOps-ing all of the things is that you can hide ugly interfacesand design exactly the interaction you want with some simple porcelain commands
  106. 106. My favorite example of this is ugliest of the ugly, Nagios.
  107. 107. [nines] hubot opened issue #4263: Nagios (229906) - fs3b/syslog - Tue Sept 25 23:40:18 PDT 2012. github/nines#4263Hubot politely delivers nagios alerts directly into chat
  108. 108. hubot nagios ack fs3b/syslog # fix stuff nagios check fs3b/syslog nagios status fs3b/syslog hubot nagios downtime fs3b/syslog 90 nagios mute fs3b/syslog nagios unmute fs3b/syslogWhich we can interact with without any unnecessary eye bleeding. Making this easy meansdevelopers and other ops engineers actually mute or schedule downtime when they’re testingthings.
  109. 109. Mobile FTWYet another awesome benefit of ChatOps is that you get mobile support for free
  110. 110. Well, that is, if you have a team of awesome iOS developers that have built an actuallyfunctioning Campfire client for the iPhoneThis lets you do anything hubot can do from your phone.Which means from your couch. Or your bed. Or a beach in Hawaii.Which means you can fix a lot of things without pulling your laptop out of your bag.
  111. 111. ChatOpsThat’s ChatOps at its finest.
  112. 112. And now for something completely differentWhile I’m showing off mobile stuff, I thought I’d slip in a demo of something else we’ve doneto make Ops more mobile friendly.
  113. 113. We’ve hacked together support for PagerDuty alerts via Apple Push Notifications. When youswipe on the alert, you go directly to the PagerDuty mobile UI for an incident
  114. 114. Which lets you ack an alert
  115. 115. while you’re still in bed
  116. 116. or on the couch.
  117. 117. BoomI can’t even begin to tell you how happy this makes me, and how less shitty it makes beingon-call
  118. 118. So, who better to summarize all of this than Hubot himself. I asked him what he thoughtabout ChatOps. Here’s what he said:
  119. 119. ChatOps all the things.Listen to what Hubot said. You’ll love it. Your ops team will love it.And you’ll help other developers learn how to interact with ops tools without any additionalwork.That’s awesome.
  120. 120. Work at GitHub jesse@github.comIf you can’t ChatOps all the things at your gig now, you could always just come work with meat GitHub.Shoot me an email if you’re interested.
  121. 121. Thanks!That’s all I have. Thanks for listening! any questions?
  122. 122. Tomorrow @ 8:30 PM Zeke’s 3rd & BrannanWhile I still have everyone’s attention, I wanted to mention the GitHub Drinkup we’rethrowing for Puppetconf again. It’s tomorrow night at 8:30pm at Zeke’s, which is on thecorner of 3rd and Brannan, everyone’s invited. I’ll see you there.Thanks again!

×