Infrastructure
testing with Jenkins,
Puppet and Vagrant

Carlos Sanchez
@csanchez
http://csanchez.org
http://maestrodev.co...
@csanchez

ASF
Member

Apache
Maven

Eclipse
Foundation

csanchez.org
maestrodev.com
How we got here
Agile

planning
iterative development
continuous integration
release soon, release often
Fear of change
Risky deployments
It works on my machine!
Siloisation
Dev Change vs. Ops stability
OPs requirements

Operating System
config files
packages installed
multi stage
configurations
dev
QA
pre-production
productio...
Deployment

How do I deploy
this?
documentation
manual steps
prone to errors
Cloud

How do I deploy
this?
to hundreds of
servers
DevOps
DevQaOps ?
DEV

QA

OPS
QA

DEV

OPS
Specs
Packages
Versions

DEV

PROD
DEV

QA

Testability
Metrics
Logs
Security
updates

DEV

PROD
is not about
the tools
but

how can I
implement
IT
Tools can
enable
change in
behavior and
eventually
change
culture
Patrick Debois
everyone is
intelligent
enough
every tool is
cloud enabled
every tool is
DevOps(tm)
3 key concepts
Continuous
Delivery
Continuous
delivery
Infrastructure as
Code
it’s all been invented, now it’s
standardized
manifests
ruby-like
ERB templates
exec { "maven-untar":
command => "tar xf /tmp/x.tgz",
cwd => "/opt",
creates => "/opt/ap...
package { 'openssh-server':
ensure => present,
}

infrastructure
IS code
service { 'ntp':
name
=> 'ntpd',
ensure
=> running,
}

declarative model
state vs process
no scripting
Follow
development
best
practices
tagging
branching
releasing
dev, QA, production

new
solutions
new
challenges
Self
servicing
Infrastructure
always
available
virtualization & cloud
empower developers
reduce time-to-market
devs buy-in
With great power
comes great
responsibility
Vagrant
empower
developers
dev-ops
collaboration
automation
Vagrant
Vagrant
Virtual and cloud automation
VirtualBox
VMWare Fusion
AWS
Rackspace

Easy Puppet and Chef provisioning
Keep VM con...
Vagrant base boxes

www.vagrantbox.es
puppet-vagrant-boxes.puppetlabs.com
anywhere! just (big) files
using Vagrant

$ gem install vagrant
$ vagrant box add centos-6.0-x86_64 
http://dl.dropbox.com/u/1627760/centos-6.0-x86_6...
Vagrant
Vagrant.configure("2") do |config|
# Every Vagrant virtual environment requires a box to build off of.
config.vm.b...
Geppetto
http://cloudsmith.github.com/geppetto
Maven and Puppet
What am I doing to automate deployment

Ant tasks plugin
ssh commands
Assembly plugin
Cargo
Capistrano
What can I do to automate deployment

Handle full deployment including infrastructure
not just webapp deployment

Help Ops...
Maven-Puppet module
A Maven Puppet module
https://github.com/maestrodev/puppet-maven
fetches Maven artifacts from the repo...
Installing Maven

$repo1 = {
id => "myrepo",
username => "myuser",
password => "mypassword",
url => "http://repo.acme.com"...
New Maven type

maven { "/tmp/maven-core-2.2.1.jar":
id => "org.apache.maven:maven-core:jar:2.2.1",
repos => ["http://repo...
New Maven type

maven { "/tmp/maven-core-2.2.1.jar":
groupId => "org.apache.maven",
artifactId => "maven-core",
version =>...
Examples
Infrastructure
www
httpd
Jenkins
Archiva

QA
httpd,
tomcat,
postgres

tomcat1
tomcat
db
postgres
Tomcat cluster + postgres
httpd
www.acme.com
tomcat
tomcat1.acme.com

tomcat
tomcat2.acme.com
postgres
db.acme.com

...
Continuous Delivery
commit

git repo

post commit hook
jenkins

developer

build

db

production

tomcat*
www

QA vm

bina...
Puppet Modules required
$ bundle install && librarian-puppet install
forge 'http://forge.puppetlabs.com'
mod 'puppetlabs/j...
mymodules/acme/manifests/db_node.pp
class 'acme::db_node' {
class { 'postgresql::server':
ip_mask_allow_all_users => '192....
mymodules/acme/manifests/tomcat_node.pp I
class acme::tomcat_node(
$db_host = 'db.local',
$repo = 'http://carlos-mbook-pro...
mymodules/acme/manifests/tomcat_node.pp II

# install maven and download appfuse war file from our archiva instance
class ...
mymodules/acme/manifests/www_node.pp
class acme::www_node($tomcat_host = 'tomcat1.local') {
class { 'apache': }
class { 'a...
manifests/site.pp

import 'nodes/*.pp'
node ‘parent’ {
class {'epel': } ->
class {'avahi':
firewall => true,
}
}
manifests/nodes/tomcat.pp

# tomcat1.acme.com, tomcat2.acme.com,...
node /tomcatd..*/ inherits ‘parent’ {
file {'/etc/motd...
manifests/nodes/qa.pp

node /qa..*/ inherits ‘parent’ {
class {'acme::db_node': }
class {'acme::tomcat_node':
db_host => '...
Infrastructure unit
testing
rspec-puppet

Unit testing your puppet manifests
Ensuring packages, config files, services,...
spec/hosts/db_spec.pp

require 'rspec-puppet'
describe 'db.acme.com' do
let(:facts) { {
:osfamily => 'RedHat',
:operatings...
spec/hosts/www_spec.pp

require 'rspec-puppet'
describe 'www.acme.com' do
let(:facts) { {
:osfamily => 'RedHat',
:operatin...
Example code and slides

Available at
http://slideshare.csanchez.org
http://github.csanchez.org
http://blog.csanchez.org
Danke!
http://csanchez.org
http://maestrodev.com

csanchez@maestrodev.com
carlos@apache.org
@csanchez
Photo Credits
Brick wall - Luis Argerich
http://www.flickr.com/photos/lrargerich/4353397797/

Agile vs. Iterative flow - Chr...
Infrastructure testing with Jenkins, Puppet and Vagrant - Agile Testing Days 2013
Infrastructure testing with Jenkins, Puppet and Vagrant - Agile Testing Days 2013
Infrastructure testing with Jenkins, Puppet and Vagrant - Agile Testing Days 2013
Infrastructure testing with Jenkins, Puppet and Vagrant - Agile Testing Days 2013
Upcoming SlideShare
Loading in...5
×

Infrastructure testing with Jenkins, Puppet and Vagrant - Agile Testing Days 2013

24,979

Published on

Extend Continuous Integration to automatically test your infrastructure.

Continuous Integration can be extended to test deployments and production environments, in a Continuous Delivery cycle, using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations, and test the infrastructure the same way continuous integration tools do with developers’ code.

Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services, … in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.

Using Vagrant, a command line automation layer for VirtualBox, we can easily spin off virtual machines with the same configuration as production servers, run our test suite, and tear them down afterwards.

We will show how to set up automated testing of an application and associated infrastructure and configurations, creating on demand virtual machines for testing, as part of your continuous integration process.

Published in: Technology

Infrastructure testing with Jenkins, Puppet and Vagrant - Agile Testing Days 2013

  1. 1. Infrastructure testing with Jenkins, Puppet and Vagrant Carlos Sanchez @csanchez http://csanchez.org http://maestrodev.com
  2. 2. @csanchez ASF Member Apache Maven Eclipse Foundation csanchez.org maestrodev.com
  3. 3. How we got here
  4. 4. Agile planning iterative development continuous integration release soon, release often
  5. 5. Fear of change Risky deployments It works on my machine! Siloisation Dev Change vs. Ops stability
  6. 6. OPs requirements Operating System config files packages installed multi stage configurations dev QA pre-production production
  7. 7. Deployment How do I deploy this? documentation manual steps prone to errors
  8. 8. Cloud How do I deploy this? to hundreds of servers
  9. 9. DevOps
  10. 10. DevQaOps ?
  11. 11. DEV QA OPS
  12. 12. QA DEV OPS
  13. 13. Specs Packages Versions DEV PROD
  14. 14. DEV QA Testability
  15. 15. Metrics Logs Security updates DEV PROD
  16. 16. is not about the tools but how can I implement IT
  17. 17. Tools can enable change in behavior and eventually change culture Patrick Debois
  18. 18. everyone is intelligent enough every tool is cloud enabled every tool is DevOps(tm)
  19. 19. 3 key concepts
  20. 20. Continuous Delivery
  21. 21. Continuous delivery
  22. 22. Infrastructure as Code it’s all been invented, now it’s standardized
  23. 23. manifests ruby-like ERB templates exec { "maven-untar": command => "tar xf /tmp/x.tgz", cwd => "/opt", creates => "/opt/apache-maven-${version}", path => ["/bin"], } -> file { "/usr/bin/mvn": ensure => link, target => "/opt/apache-maven-${version}/bin/mvn", } file { "/usr/local/bin/mvn": ensure => absent, require => Exec["maven-untar"], } file { "$home/.mavenrc": mode => "0600", owner => $user, content => template("maven/mavenrc.erb"), require => User[$user], }
  24. 24. package { 'openssh-server': ensure => present, } infrastructure IS code
  25. 25. service { 'ntp': name => 'ntpd', ensure => running, } declarative model state vs process no scripting
  26. 26. Follow development best practices tagging branching releasing dev, QA, production new solutions new challenges
  27. 27. Self servicing
  28. 28. Infrastructure always available virtualization & cloud empower developers reduce time-to-market
  29. 29. devs buy-in With great power comes great responsibility
  30. 30. Vagrant empower developers dev-ops collaboration automation
  31. 31. Vagrant
  32. 32. Vagrant Virtual and cloud automation VirtualBox VMWare Fusion AWS Rackspace Easy Puppet and Chef provisioning Keep VM configuration for different projects Share boxes and configuration files across teams base box + configuration files
  33. 33. Vagrant base boxes www.vagrantbox.es puppet-vagrant-boxes.puppetlabs.com anywhere! just (big) files
  34. 34. using Vagrant $ gem install vagrant $ vagrant box add centos-6.0-x86_64 http://dl.dropbox.com/u/1627760/centos-6.0-x86_64.box $ $ $ $ $ $ vagrant vagrant vagrant vagrant vagrant vagrant init myproject up ssh suspend resume destroy
  35. 35. Vagrant Vagrant.configure("2") do |config| # Every Vagrant virtual environment requires a box to build off of. config.vm.box = "CentOS-6.4-x86_64-minimal" config.vm.box_url = "https://.../CentOS-6.4-x86_64-minimal.box" # web server config.vm.define :www do |config| config.vm.hostname = "www.acme.local" config.vm.network "forwarded_port", guest: 80, host: 10080 config.vm.network "private_network", ip: "192.168.33.12" end config.vm.provision :puppet do |puppet| puppet.module_path = "modules" puppet.manifest_file = "site.pp" end end
  36. 36. Geppetto
  37. 37. http://cloudsmith.github.com/geppetto
  38. 38. Maven and Puppet
  39. 39. What am I doing to automate deployment Ant tasks plugin ssh commands Assembly plugin Cargo Capistrano
  40. 40. What can I do to automate deployment Handle full deployment including infrastructure not just webapp deployment Help Ops with clear, automated manifests Ability to reproduce production environments in local box using Vagrant / VirtualBox / VMWare Use the right tool for the right job
  41. 41. Maven-Puppet module A Maven Puppet module https://github.com/maestrodev/puppet-maven fetches Maven artifacts from the repo manages them with Puppet no more extra packaging
  42. 42. Installing Maven $repo1 = { id => "myrepo", username => "myuser", password => "mypassword", url => "http://repo.acme.com", } # Install Maven class { "maven::maven": version => "2.2.1", } -> # Create a settings.xml with the repo credentials class { "maven::settings" : servers => [$repo1], }
  43. 43. New Maven type maven { "/tmp/maven-core-2.2.1.jar": id => "org.apache.maven:maven-core:jar:2.2.1", repos => ["http://repo1.maven.apache.org/maven2", "http://mirrors.ibiblio.org/pub/mirrors/maven2"], }
  44. 44. New Maven type maven { "/tmp/maven-core-2.2.1.jar": groupId => "org.apache.maven", artifactId => "maven-core", version => "2.2.1", packaging => "jar", repos => ["http://repo1.maven.apache.org/maven2", "http://mirrors.ibiblio.org/pub/mirrors/maven2"], }
  45. 45. Examples
  46. 46. Infrastructure www httpd Jenkins Archiva QA httpd, tomcat, postgres tomcat1 tomcat db postgres
  47. 47. Tomcat cluster + postgres httpd www.acme.com tomcat tomcat1.acme.com tomcat tomcat2.acme.com postgres db.acme.com ...
  48. 48. Continuous Delivery commit git repo post commit hook jenkins developer build db production tomcat* www QA vm binary repository integration testing
  49. 49. Puppet Modules required $ bundle install && librarian-puppet install forge 'http://forge.puppetlabs.com' mod 'puppetlabs/java', '>=1.0.0' mod 'puppetlabs/apache', '>=0.9.0' mod 'puppetlabs/postgresql', '>=3.0.0' mod 'puppetlabs/firewall' mod 'camptocamp/tomcat', :git => 'https://github.com/carlossg/puppet-tomcat.git', :ref => 'setclasspath' mod 'maestrodev/maven', '>=1.0.0' mod 'stahnma/epel', '>=0.0.2' mod 'maestrodev/avahi', '>=1.0.0' mod 'acme', :path => 'mymodules/acme'
  50. 50. mymodules/acme/manifests/db_node.pp class 'acme::db_node' { class { 'postgresql::server': ip_mask_allow_all_users => '192.168.0.0/0', listen_addresses => '*', postgres_password => 'postgres', } -> postgresql::server::db { 'appfuse': user => 'appfuse', password => 'appfuse', grant => 'all', } firewall proto port action } } { '100 allow postgres': => 'tcp', => '5432', => 'accept',
  51. 51. mymodules/acme/manifests/tomcat_node.pp I class acme::tomcat_node( $db_host = 'db.local', $repo = 'http://carlos-mbook-pro.local:8000/repository/all/', $appfuse_version = '2.2.2-SNAPSHOT', $service = 'tomcat-appfuse', $app_name = 'appfuse', $tomcat_root = '/srv/tomcat') { # install java class { 'java': } # install tomcat class { 'tomcat': } -> # create a tomcat server instance for appfuse # It allows having multiple independent tomcat servers in different ports tomcat::instance { 'appfuse': ensure => present, http_port => 8080, } # where the war needs to be installed $webapp = "${tomcat_root}/${app_name}/webapps/ROOT"
  52. 52. mymodules/acme/manifests/tomcat_node.pp II # install maven and download appfuse war file from our archiva instance class { 'wget': } -> class { 'maven::maven' : version => '3.0.4', } -> # clean up to deploy the next snapshot exec { "rm -rf ${webapp}*": } -> maven { "${webapp}.war": id => "org.appfuse:appfuse-spring:${appfuse_version}:war", repos => [$repo], require => File["${tomcat_root}/${app_name}/webapps"], notify => Service[$service], } -> firewall proto port action } } { '100 allow tomcat': => 'tcp', => '8080', => 'accept',
  53. 53. mymodules/acme/manifests/www_node.pp class acme::www_node($tomcat_host = 'tomcat1.local') { class { 'apache': } class { 'apache::mod::proxy_http': } # create a virtualhost that will proxy the tomcat server apache::vhost { "${::hostname}.local": port => 80, docroot => '/var/www', proxy_dest => "http://${tomcat_host}:8080", } firewall { '100 allow apache': proto => 'tcp', port => '80', action => 'accept', } }
  54. 54. manifests/site.pp import 'nodes/*.pp' node ‘parent’ { class {'epel': } -> class {'avahi': firewall => true, } }
  55. 55. manifests/nodes/tomcat.pp # tomcat1.acme.com, tomcat2.acme.com,... node /tomcatd..*/ inherits ‘parent’ { file {'/etc/motd': content => ”tomcat server: ${::hostname}n”, } class {'acme::tomcat_node'} }
  56. 56. manifests/nodes/qa.pp node /qa..*/ inherits ‘parent’ { class {'acme::db_node': } class {'acme::tomcat_node': db_host => 'localhost', } class {'acme::www_node': tomcat_host => 'localhost', } }
  57. 57. Infrastructure unit testing
  58. 58. rspec-puppet Unit testing your puppet manifests Ensuring packages, config files, services,...
  59. 59. spec/hosts/db_spec.pp require 'rspec-puppet' describe 'db.acme.com' do let(:facts) { { :osfamily => 'RedHat', :operatingsystem => 'CentOS', :operatingsystemrelease => ‘6.3’} } it { should contain_class('postgresql::server') } end
  60. 60. spec/hosts/www_spec.pp require 'rspec-puppet' describe 'www.acme.com' do let(:facts) { { :osfamily => 'RedHat', :operatingsystem => 'CentOS', :operatingsystemrelease => ‘6.3’} } it { should contain_package('httpd') } end
  61. 61. Example code and slides Available at http://slideshare.csanchez.org http://github.csanchez.org http://blog.csanchez.org
  62. 62. Danke! http://csanchez.org http://maestrodev.com csanchez@maestrodev.com carlos@apache.org @csanchez
  63. 63. Photo Credits Brick wall - Luis Argerich http://www.flickr.com/photos/lrargerich/4353397797/ Agile vs. Iterative flow - Christopher Little http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg DevOps - Rajiv.Pant http://en.wikipedia.org/wiki/File:Devops.png Pimientos de Padron - Howard Walfish http://www.flickr.com/photos/h-bomb/4868400647/ Compiling - XKCD http://xkcd.com/303/ Printer in 1568 - Meggs, Philip B http://en.wikipedia.org/wiki/File:Printer_in_1568-ce.png Relativity - M. C. Escher http://en.wikipedia.org/wiki/File:Escher%27s_Relativity.jpg Teacher and class - Herald Post http://www.flickr.com/photos/heraldpost/5169295832/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×