Continuously Testing
Infrastructure
Puppet Conf, San Francisco, 2014
Gareth Rushgrove
Beyond Module Testing
@garethr
Gareth Rushgrove
Gareth Rushgrove
Gareth Rushgrove
Not talking about
Finished software
Gareth Rushgrove
Testing individual modules
Gareth Rushgrove
puppet-lint, puppet-syntax,
rspec-puppet, beaker
Gareth Rushgrove
Gareth Rushgrove
Am talking about
Experiments
Gareth Rushgrove
Testing images and
containers
Gareth Rushgrove
Test driving infrastructure
as a service
Gareth Rushgrove
Testing with PuppetDB
Gareth Rushgrove
Testing
images and
containers
1
Gareth Rushgrove
Packer builds images
based on a JSON
template
Gareth Rushgrove
Gareth Rushgrove
It has some Puppet
integration too
Gareth Rushgrove
Gareth Rushgrove
But how do we know the
image works?
Gareth Rushgrove
Lets add some tests!
Gareth Rushgrove
Gareth Rushgrove
shaunduncan/packer-provisioner-host-command
Gareth Rushgrove
serverspec.org
Gareth Rushgrove
Gareth Rushgrove
Gareth Rushgrove
Gareth Rushgrove
Serverspec also supports
port, file, ppa, selinux,
user, group, lxc, iptables,
cron and more
Gareth Rushgrove
Only publish the image if
the tests pass
Gareth Rushgrove
Run tests automatically
with a continuous
integration system
Gareth Rushgrove
Gareth Rushgrove
Gareth Rushgrove
garethr/packer-serverspec-example
Gareth Rushgrove
Gareth Rushgrove
Same approach works
with containers too
Gareth Rushgrove
Gareth Rushgrove
garethr/docker-spec-example
Gareth Rushgrove
Test drive
your IaaS
2
Test driven development
Gareth Rushgrove
First the developer writes
an automated test case
that defines a desired
improvement or new
function
Gareth Rushgrove
Then produces the
minimum amount of code
to pass that test
Gareth Rushgrove
And finally refactors the
new code
Gareth Rushgrove
Gareth Rushgrove
First the developer writes
an automated test case
that defines a desired
improvement or new
function
Your infrastructure should!
have an API
Gareth Rushgrove
What if we write
assertions against!
that API?
Gareth Rushgrove
Aside: Clojure
2.1
Gareth Rushgrove
Great for building DSLs
Gareth Rushgrove
Don’t worry, you could
write the examples in any
language
Gareth Rushgrove
Policy driven development
Gareth Rushgrove
I don’t want to launch too
many nodes, they’re
expensive
Gareth Rushgrove
Policy
Gareth Rushgrove
I don’t want any stopped
nodes, they are costing
me money
Gareth Rushgrove
Policy
Gareth Rushgrove
Large nodes are really
expensive, so limit their
usage
Gareth Rushgrove
Policy
Gareth Rushgrove
We should be backing up
every node
Gareth Rushgrove
Policy
Gareth Rushgrove
I only want nodes in
London and !
San Francisco
Gareth Rushgrove
Policy
Gareth Rushgrove
All our nodes should be
named environment-name
Gareth Rushgrove
Policy
Gareth Rushgrove
garethr/digitalocean-expect
Gareth Rushgrove
Gareth Rushgrove
Now we have the tests,
we can provision some
infrastructure
Gareth Rushgrove
Aside: Provisioning
with Puppet
2.2
Gareth Rushgrove
Gareth Rushgrove
puppetlabs/gce_compute
Gareth Rushgrove
Gareth Rushgrove
Gareth Rushgrove
garethr/digitalocean
Gareth Rushgrove
Gareth Rushgrove
bobtfish/aws_api
Gareth Rushgrove
Testing with
PuppetDB
3
Aside: PuppetDB
3.1
puppetlabs/puppetdb
Gareth Rushgrove
PuppetDB can store a lot
of data about your
infrastructure
Gareth Rushgrove
The most recent facts
from every node
Gareth Rushgrove
The most recent catalog
for every node
Gareth Rushgrove
A wide range of metrics
Gareth Rushgrove
Gareth Rushgrove
I want to run the same
operating system on all
hosts
Gareth Rushgrove
Policy
Gareth Rushgrove
Security enforcing
packages should be
installed everywhere
Gareth Rushgrove
Policy
Gareth Rushgrove
I want to limit how many
puppet resources I’m
using
Gareth Rushgrove
Policy
Gareth Rushgrove
We should avoid heavy I/
O load on the database by
maintaining a high catalog
duplication rate
Gareth Rushgrove
Policy
Gareth Rushgrove
garethr/puppetdb-expect
Gareth Rushgrove
Testing based on
PuppetDB
3.2
PuppetDB is a great
source of context for tests
Gareth Rushgrove
Generate serverspec tests
from PuppetDB data
Gareth Rushgrove
Automatically detect
hosts, and generate
commands
Gareth Rushgrove
Gareth Rushgrove
Match puppet resources to
serverspec resources
Gareth Rushgrove
Gareth Rushgrove
For instance on a Puppet
Enterprise master
Gareth Rushgrove
Gareth Rushgrove
Run serverspec tests on
all puppet managed hosts
Gareth Rushgrove
Gareth Rushgrove
garethr/serverspec-puppetdb
Gareth Rushgrove
Conclusions
Is this monitoring?
Gareth Rushgrove
We’re still moving towards
infrastructure as code
Gareth Rushgrove
Infrastructure as code
rather than infrastructure
from code
Gareth Rushgrove
Taking about policy as
code might help
communicate intent
Gareth Rushgrove
Questions?
And thanks for listening

Continuously Testing Infrastructure - Beyond Module Testing - PuppetConf 2014