Islands: Puppet at Bulletproof Networks

Islands

Who arethese guys?

Mick Pollard @aussielunix &Lindsay Holmwood @auxesis

Puppet usersfor > 5 years

BULLETPROOF

IaaS &ManagedServices

Movember Australian Museum Rebel Group Blackmores Angus & Robertson Telstra Perisher BlueScope Steel Woolworths DMG Radio Clive Peters Deloitte Clemenger budget.gov.au Nissan AOC Nova Sydney Airports Whirlpool Smooth Theiss Borders Fosters Country Road Midas Australian Geographic SensisTourism Victoria ABS AusPost DET FWA Vodafone

Using Puppet since 2008

Uniquechallenges

Strong isolation

Networksegregationwith VLANs

CentralPuppetmasterisnʼt an option

Thoroughchange control

Rapid growth

How do weuse Puppet?

Standalone systems(puppetmaster-less)

Internalinfrastructure

Full customerenvironments

Standalone systems(puppetmaster-less)

Campaign drivenbusiness

budget.gov.au

movember.com

mamamia.com.au

Reverseproxies

Nginxwith customisation

More detail inJohn Ferlitoʼs talk at 14.00

Internalinfrastructure

Vanilla

OnePuppetmaster

Ubuntu(Lucid or Precise)

Full customerenvironments

Every customer has their own puppetmaster

“Islands of Puppet”

Copypasta

Conﬁguration drift

apache

apachecustomer-a

apache customer-bcustomer-a

apache customer-bcustomer-a customer-c

apache customer-bcustomer-a customer-d customer-c

Poor code share

What if customersedit the code?

How do we maintaincommon code?

Commonalities

Mix of Puppetversions

0.25(as provided by Ubuntu)

2.7(as provided by Puppet Labs)

Mix ofOperatingSystems

lucid precise2.7 internal infrastructure some customers0.25 most customers

Passenger > webrick

--no-daemonize

Default behaviour is orthogonal tochange control

We dont wantsystems to change without control

All changesinitiated byan engineer

nodes + roles

node stlyqy-lvs02.cust.bulletproof.net { server { $fqdn: } include snmp::server::lvs include sysctl::lvs include keepalive::lvs include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan include ript}

define server($collectd_client_report_to=collectd.bulletproof.net) { include motd include augeas include apt include utils include puppet::client include ssh::server include ssh::authorized_keys include ntp::client include postfix::satellite include ruby::dev include ruby::rubygems include bzr::client include git::common include git::github include snmp::server include vmware::tools include apparmor::disable collectd::client { "${fqdn}": report_to => $collectd_client_report_to }}

app_serverdatabase_serverfile_servermanagement_servermemcache_servermonitor_serverproxy_serverpuppetmaster_serverredis_serversingle_serversphinx_serverstatic_server

Heirais the future

We useCapistrano

sshin-a-parallel-for-loop

Why cap andnot mcollective?

We deployeverything with cap

Monitoring conﬁguration Firewall conﬁguration Web applications Internal tools

Consistentdeployment toolacross all projects

Principle ofleast surprise

Engineers learn 1 tool

Puppet is no different to therest of our stack

How do we usecap + Puppet?

Puppet changes

cap puppet:go ROLES=lvs options="--noop"cap puppet:go ROLES=lvs

Smoke tests

$ cap puppet:go options="--noop"# ...infmon hosts serves a Nagios page over httpsFinished in 1.67 seconds1 example, 0 failures

Works out all rolesthat hosts within a run belong to

Runs tests tagged with those roles

Fast feedback onchange success

Bootstrapping

cap node:bootstrap HOSTFILTER=lvs-08.bp.net

Takes VM inunknown state

Brings intoknown state forPuppet 2.7 run

Limitations

Singling outhosts is tricky

Re-using data across commandsrequires... creativity

servers = []run "mysql -e "SHOW MASTER STATUS;" | tail -n 1" do |channel, type, data| hostname = channel[:host] filename = data.split(/s+/).first position = data.split(/s+/).last servers << { :hostname => hostname, :filename => filename, :position => position }end

ROLES andHOSTFILTER

There areedge cases

It does the job

How have wetried to solve them?

First iteration

Modules

Stored on GitHub

Drink from the ﬁrehose

puppet-module-tool

github.compuppetlabs/puppet-module-tool

gem install puppet-module

Modulefile

puppet-module build

bulletproofnetworks-apache-1.3.0.tar.gz

Puppet forge

Publicby default

Not great if you dontwant to open sourceall your secret sauce

But puppet-module-tool isinteresting...

...can we fake the forge?

Pain points

Arduous releaseworkﬂow

Better suitedfor infrequent changes

High barrier ofentry for customers to submit patches

Sharing bugﬁxes & improvementsrequires signiﬁcant refactoring

Limitedreporting oncustomer lag

Second iteration

Bundler

Gemfile

#!/usr/bin/env rubysource :rubygemsgem capistrano, 2.9.0gem capistrano-ext, 1.2.1gem colorizegem puppet, 2.7.13gem puppet-modulegroup :test do gem rspec, 2.8.0 gem mechanize gem puppet-lintend

Can wereuse it?

Tim Sharpe @ GitHub(@rodjek)

Messy working prototype

pre-alpha quality

More research required

Librarian

“A framework for bundlers”

librarian-chefmanages Chef repositories

by Jay Feldblum github.comapplicationsonline/librarian

librarian-puppet

“You can all stop using git submodules now”

gem install librarian-puppet --pre

github.comrodjek/librarian-puppet

Bundler-like behavior for Puppet

Puppetfile

#!/usr/bin/env rubyforge "http://forge.puppetlabs.com"mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"

#!/usr/bin/env rubyforge "http://forge.puppetlabs.com"mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"

#!/usr/bin/env rubyforge "http://forge.puppetlabs.com"mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"mod "apt", :git => "git://github.com/puppetlabs/puppetlabs-apt.git" :ref => feature/master/dans_refactor

One canonical module

modules/ in.gitignore

librarian-puppet outdated tells you what needs to be updated

librarian-puppet updateupdates your modules

Demo!http://aussielunix.github.com/jenkins-appliance/

Policy

Always use a modules master

Make changesto modules as usual

git commit && git push the module

Make modulechanges generic by default

Only brancha module when:

1. something is super client speciﬁc

2. there is anunmissable deadline

3. testing ideas

Set a reminder tomerge changes into master

Use pull requests on GitHub fordangerous changes

Take aways:

Puppet modules are pretty neat

Keep feedback loops short

Code share is king

Do you have similar problems?

How do yousolve them?

Thank you!

Islands: Puppet at Bulletproof Networks

by Lindsay Holmwood on Jun 07, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

Islands: Puppet at Bulletproof Networks Presentation Transcript