Puppet at GitHub - PuppetConf 2013

@wfarr
GitHub
Operations
Known Aliases:
King of Kebabs
The Chairman
Mr. Caremad !

The State of Puppet at GitHub
"

" The State of Puppet at GitHub
github/boxen

~1.5 years old

open-sourced ~7 months ago

~240 open-source puppet modules

puppet 3.latest

supports ruby 1.8.7, 1.9.3, 2.0.0

hiera

OS X

Linux support in-progress

#
tomorrow @ 2:20pm

github/puppet

$

~5 years old

0.24.x — 2.7.x

121 modules

~280k lines of code

every single employee has commit access

15.5k commits to master past 12 months

by 86 contributors

% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% %

% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% %
42.5%

% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% %
now with ops taken out

0
100
200
300
400
2012-08-25
2012-10-20
2012-12-15
2013-02-09
2013-04-06
2013-06-01
2013-07-27
commits on master per week, last 12 months

0
1000
2000
3000
4000
total commits by author past year

0
175
350
525
700
total commits by author past year, except ops

0
175
350
525
700
total commits by authors with >10 commits, past year, except ops

single puppetmaster

rubygems

ruby 1.8.7

unicorn

puppet 2.7.latest

~600 nodes

run hourly via crond

puppetdb

nagiosdb

"yo puppetdb, gimme all your
nagios::object::* resources so I can
update this nagios conﬁg"

ﬁltergendb

"yo puppetdb, gimme all the
filtergen::rule resources I would realize
so I can update this filtergen config"

⚡ puppetdb ⚡

as it turns out, an api call is faster
than running puppet on a host

gpanel

"Imagine Puppet Dashboard meets
Razor and went on a weekend trip to the
beach with the Heroku API and drank
epic amounts of blue drink"

aka we reinvented our own, smaller
version of Foreman

inventory

app conﬁguration

versioning of conﬁguration values

Create nil => 1
Update 1 => 2
Delete 2 => nil

let's make credential rolling less awful

provisioning

provisioning is typically awful

we sprinkled in some ChatOps

a little bit later...

enc

we have the dumbest ENC out there

⋆

# /usr/local/sbin/fetch_gpanel_enc
fe1.rs.github.com
---
parameters:
gpanel_cabinet: D20-13
gpanel_enabled: true
gpanel_monitored: false

we never specify classes via the ENC

any variables we pass through
are preﬁxed with gpanel_

( How GitHub Writes Puppet
rodjek/puppet-lint

if you aren't using puppet-lint
to audit your puppet codebase,
you are doing it wrong

puppet-lint enforces the
Puppet Labs style guide

puppet-lint can even ﬁx a ton
of linter errors for you

put it in a pre-commit hook

$ git commit -am "can't lint this"
modules/github/manifests/role/redis.pp: syntax ok
modules/github/manifests/role/redis.pp - WARNING: => is not
properly aligned on line 118
1 errors found, aborting commit.

and then buy rodjek a beer

rodjek/rspec-puppet

if you aren't writing tests for
your puppet code before
running it on a server,
you are doing it wrong

use whatever framework/library

rspec-puppet just happens to be
a pretty good one

$ git commit -am "tests dont pass but whatever lol"
1) Expected redis::server would include class
"more_than_a_single_c_thread"
1 failures encountered, aborting commit.

and then buy rodjek another beer

node deﬁnitions

we don't use an ENC to describe
node classes

node /^github-redisd+/ {
class { 'github::role::polling_redis':
enabled => $::gpanel_enabled,
environment => $::gpanel_environment,
private_ipv4 => $::ipaddress,
}
}

abstractions all the way down

treat your site classes as cascades
down to your dist classes

class redis::server(
# params
) {
class { 'redis::config': ... }
-> class { 'redis::package': ... }
~> class { 'redis::service': ... }
}

class github::redis(
# params
) inherits github::defaults {
$memory = $environment ? {
'stg' => '2G',
default => $half_memory_gb
}
class { 'redis::server': ... }
}

class github::role::polling_redis(
# params
) {
class { 'github::redis':
# overrides based on specific node
}
}

augeas

you know what's not awesome?

an erb template that requires your
class to take 52876423 parameters
so you can conﬁgure every possible
value in my.cnf

augeas { 'my.cnf/performance':
context => '/files/etc/mysql/my.cnf/mysqld',
changes => [
# automatic dump/restore
'set innodb_auto_lru_dump 18000',
# innodb
"set innodb_buffer_pool_size ${innodb_buffer_pool_size}",
'set innodb_log_file_size 256M',
'set innodb_log_buffer_size 8M',
'set innodb_lazy_drop_table 1',
],
require => Percona::Server[$::fqdn]
}

it can seem complex and scary

it is an amazing tool

) How GitHub Ships Puppet
continuous integration

commit gets pushed

jenkins runs the test suite

status gets posted back to GitHub.com

tmm1/test-queue

remembers how to better parallelize tests

continuous deployment

tests passed for commit, Hubot auto-deploys

branch deploy everything

a lot of shops have a few environments

testing
staging
production

currently we have 181 environments

that's not the same as 181 nodes
not running production

we only have 5 of those

Hubot automatically merges the
master branch before allowing any
branch to deploy

ChatOps

202 employees
100% access to Puppet
100% trust

the list of roles you'd never think would touch production puppet
webkit developer
core git developers
core ruby developers
svn developers
search developer
frontend designers
windows developers
mac developers
support
enterprise sales

every puppet run happens in chat

all puppet output goes to chat

everyone can see everything

everyone can do anything

hands-on learning, by accident

eliminate disruptive questions

The Future of Puppet at GitHub
*

* The Future of Puppet at GitHub
puppet 3.x

upgrading a large, old codebase
from 2.7 to 3 is really painful

we've been talking about it for a year

we're still trying to push forward, but...

helping maintain security ﬁxes for 2.7

goal is to be on 3.2 by end of year

ruby 2.0

moar faster rubby

we get this for free when we move to 3

mcollective

aka the story of github/shell and the
wonders of ssh in a for-loop

there are a lot of cool things about
mcollective

I think it will be a part of our stack in
the future

but you can still go pretty damn far
with bash and ssh

even more puppetdb tooling

waiting for multiple runs to converge
exported resources is painful

use the puppetdb API to skip all that

even more gpanel integration

"databags"

"node search"

steal good ideas from other tools

"all software is terrible"
— anyone who's worked with software long enough

"no software is better than no software"
— rtomayko

the plumbing doesn't matter when all you
care about is the porcelain

write some damned good porcelain

HE IS ALSO NOT MY BROTHER,
BUT YOU REALLY SHOULD BUY
HIM A BEER

YOU SHOULD BUY THIS
MAN A LOT OF BEER

THANKS
https://speakerdeck.com/wfarr/
puppet-at-github-puppetconf-2013
!

Puppet at GitHub - PuppetConf 2013

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to Puppet at GitHub - PuppetConf 2013

Similar to Puppet at GitHub - PuppetConf 2013 (20)

More from Puppet

More from Puppet (20)

Recently uploaded

Recently uploaded (20)

Puppet at GitHub - PuppetConf 2013