Deployment Tactics
Upcoming SlideShare
Loading in...5

Deployment Tactics



The slides from my Deployment Tactics talk at the ThinkVitamin Code Management online conference (

The slides from my Deployment Tactics talk at the ThinkVitamin Code Management online conference (



Total Views
Views on SlideShare
Embed Views



1 Embed 2 2



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Deployment Tactics Deployment Tactics Presentation Transcript

  • LOYM ENTDEP ICS TACT Managing code fromdevelopment to productionIan Barber - |
  • Image:
  • - Table of Contents -1.... Change Control2.... Environments3.... Version Control4.... The Deploy Process5.... Scripts6.... Continuous Integration7.... Remote Releases8.... Packaged Releases9.... Package Management10.. Managing Hotfixes11.. Managing Database Changes12.. Rollbacks13.. Tactical Deployment
  • Change control plan execute change changeidentify verify need close deliver
  • export copy to code serverrequire comparerelease md5 report restart back apache
  • Change Request FormRequested By: J. Teamlead Authorised By: S. ManagerSubmit Date: 2011-01-27 Change Date: 2011-02-04Reason For Change:Resolve JIRA-1602 - Listen for new .com variants on vhostChange Request:Release tag 1.1.3 via normal processmv /etc/httpd/conf.d/fooweb.conf /etc/httpd/conf.d/fooweb.oldmv ~releases/1.1.3/conf/fooweb.conf /etc/httpd/conf.d/fooweb.confVerification: shows the same page as http://fooweb.comRollback:Re-release 1.1.2mv /etc/httpd/conf.d/fooweb.old /etc/httpd/conf.d/fooweb.conf
  • Environments Production Developmentstatic verboserobust dynamicreliable unstableoptimised experimental
  • on Envir onmentThe Producti Image:
  • The Staging EnvironmentImage:
  • n Enviro nment The Int egratioImage:
  • The Developme nt Environmen t Image:
  • /branches/newpage /branches/... /branches/search/trunk /branches/1.1.2 /tags/1.1.2
  • Development /branches/newpage /branches/... /branches/search/trunkIntegration /branches/1.1.2 /tags/1.1.2 Staging Production
  • master release1.1.1devel search feature long feature
  • Productionmaster release1.1.1Stagingdevel search feature Integration long feature Development
  • The DEPLoy PROCESStransparent flexible easy scalable graceful reliable
  • support SMTP process config apache vhost appupdate code config file perms packages commandslibpngupdate cache restart service service
  • code config packagerepository repository repository ta daserver commands deployment server server controller
  • code config packagerepository repository repository da taserver commands deployment server + data controller server
  • BUILD SCRIPTS#!/bin/bash# Deployment script for FooWeb Projectgit archive --format=tar --remote=git:// HEAD -o fooweb.tartar -xf fooweb.tar /var/wwwservice httpd restart
  • #!/bin/bash# Deployment script for FooWeb Projectsvn export svn://localhost/fooweb-service/trunk releasecd release && mkdir buildcp -r web/* build/javac -cp /usr/share/java/servlet-api-2.5.jar -d build/WEB-INF/classes src/com/fooweb/service/*.javacd build && jar cvf ../fooweb.war * && cd ../# assumes autoDeploy is truecp fooweb.war /var/lib/tomcat6/webapps
  • BUILDS TOOLS tests releasecode build docs test assets results
  • buildtools
  • <?xml version="1.0" encoding="UTF-8"?><project name="FooWeb"><property name="install" location="/var/lib/tomcat6/webapps" /><property name="svn.repo" value="svn://localhost/fooweb-service/trunk" /><!--A "clean" target to delete compiledfiles--><target name="clean"> <delete dir="build" /> <delete dir="release" /> <delete file="fooweb.war" /></target>
  • <!-- Checkout, mkdir and compile--><target name="build"> <exec executable="svn"> <arg line="export ${svn.repo}release" /> </exec> <mkdir dir="build"/> <copy todir="build"> <fileset dir="release/web" /> </copy> <javac srcdir="release/src"destdir="build/WEB-INF/classes/"> <classpath> <pathelement path="/usr/share/java/servlet-api-2.5.jar"/> </classpath> </javac></target>
  • <!-- Build our WAR file --><target name="war" depends="build"> <war destfile="fooweb.war" webxml="build/WEB-INF/web.xml"> <fileset dir="build"/> <classes dir="build/WEB-INF/classes"/> </war></target><!-- Copy our file --><target name="deploy" depends="war"> <copy file="fooweb.war" todir="${install}" /></target></project>
  • $ sudo ant deployBuildfile: build.xmlbuild: [exec] Exported revision 8. [mkdir] Created dir: /tmp/build [copy] Copying 2 files to /tmp/build [copy] Copied 3 empty directories to 1 empty directory under /tmp/build [javac] Compiling 1 source file to /tmp/build/WEB-INF/classeswar: [war] Building war: /tmp/fooweb.wardeploy: [copy] Copying 1 file to /var/lib/tomcat6/webappsBUILD SUCCESSFUL Total time: 2 seconds
  • CONTINUOUS INTEGRATION Look at the Hud son Wiki at dson-ci.org
  • <project name="Fooweb" default="build"> <target name="build" depends="phpunit" /> <target name="init"> <mkdir dir="${basedir}/build/logs" /> </target> <target name="phpunit" depends="init"> <exec executable="phpunit" dir="${basedir}/tests" failonerror="on"> <arg line=" --log-junit ${basedir}/build/logs/phpunit.xml --coverage-clover ${basedir}/build/logs/clover.xml --coverage-html ${basedir}/build/logs/coverage" /> </exec> </target>
  • <target name="phpcpd" depends="init"> <exec executable="phpcpd" dir="${basedir}/application" failonerror="on"> <arg line=" --log-pmd ${basedir}/build/logs/php-cpd.xml ." /> </exec> </target></project>
  • server server authorized_keys authorized_keys cp ss /s h /s h ss cp deployment controller id_rsa.pubuser “deploy” ssh-keygen -t rsa
  • Fabricfrom fabric.api import * http://fabfile .org# Development environmentdef dev(): env.user = deployer env.roledefs = { "web" : [localhost], "db" : [localhost], }# Production environmentdef production(): env.user = deployer env.roledefs = { "web" : [,], "db" : [], }
  • # Package up release - run localdef prepare_deploy(): local(svn export svn://localhost/fooweb/trunk release) with cd(release): local(tar cvzf ../fooweb.tar.gz .) local(rm -rf release)# Restart web server@roles(web)def restart_webserver(): sudo(/etc/init.d/apache2 restart)
  • # Deploy to remote servers@roles(web)def deploy(): prepare_deploy() # in case of already existing with settings(warn_only=True): run(mkdir /tmp/release) run(rm -rf /tmp/release/*) put("fooweb.tar.gz", /tmp/release) with cd(/tmp/release): run("tar xvzf fooweb.tar.gz") run("rm -rf fooweb.tar.gz") run("mv * /tmp/test") restart_webserver(); local("rm -rf fooweb.tar.gz");
  • $ fab dev deploy[localhost] run: svn export svn://localhost/fooweb/trunk release[localhost] run: tar cvzf ../fooweb.tar.gz .[localhost] run: rm -rf release[localhost] run: mkdir /tmp/release[localhost] err: mkdir: cannot createdirectory `/tmp/release: File existsWarning: run() encountered an error (returncode 1) while executing mkdir /tmp/release[localhost] run: rm -rf /tmp/release/*[localhost] put: fooweb.tar.gz -> /tmp/release/fooweb.tar.gz
  • [localhost] run: tar xvzf fooweb.tar.gz[localhost] run: rm -rf fooweb.tar.gz[localhost] run: mv * /tmp/test[localhost] sudo: /etc/init.d/apache2 restartPassword for ianbarber@localhost:[localhost] out: * Restarting web serverapache2[localhost] out: ... waiting ...done.[localhost] run: rm -rf fooweb.tar.gzDone.Disconnecting from localhost... done.
  • $ fab production deploy[localhost] run: tar cvzf ../fooweb.tar.gz .....[] run: rm -rf /tmp/release/[localhost] run: tar cvzf ../fooweb.tar.gz .....[] run: mkdir /tmp/release....Disconnecting from from done
  • $ mkdir config && cd config && capify .[add] writing ./Capfile[add] making directory ./config[add] writing ./config/deploy.rb[done] capified!set :application, "set your application name "set :repository, "set your repository"set :scm, :subversionrole :web, "your web-server here"role :app, "your app-server here"role :db, "your primary db-serverhere", :primary => true Capi stranorole :db, "slave db" .com/ https:/ /github capis trano/
  • set :application, "fooweb"set :repository,"svn://localhost/fooweb/trunk"set :scm, :subversionset :scm_username, "deployment"set :scm_password, "s3kkr3tp4a55"set :scm_checkout, "export"set :keep_releases, 4set :normalize_asset_timestamps, falseset :deploy_to, "/usr/local/#{application}"role :web, ""role :web, ""role :db, ""
  • namespace :deploy do task :migrate do # nothing end task :restart do sudo "/etc/init.d/apache2 restart" endendnamespace :fooweb do task :perms do sudo "chmod -R a+w #{deploy_to}" endendafter "deploy:setup", "fooweb:perms"
  • $ cap deploy:setup * executing `deploy:setup * executing "sudo mkdir -p /usr/local/fooweb [...]" servers: ["primary","secondary", "backend"] [backend] executing command [...] command finished triggering after callbacks for deploy:setup * executing `fooweb:perms * executing "sudo chmod -R a+w /usr/local/fooweb" servers: ["primary","secondary","backend"] [primary] executing command [...] command finished
  • $ cap deploy * executing `deploy * executing `deploy:update ** transaction: start * executing `deploy:update_code executing locally: "svn info svn://localhost/fooweb/trunk -rHEAD"/usr/bin/svn * executing "svn checkout -q -r17 svn://localhost/fooweb/trunk /usr/local/fooweb/releases/20110116192456 && (echo 17 > /usr/local/fooweb/releases/20110116192456/REVISION)" servers: [""] [] executing command[....] * executing `deploy:finalize_update * executing "chmod -R g+w /usr/local/fooweb/
  • /usr/local/fooweb/!"" current -> releases/20110116192316!"" releases#   !"" 20110116190608#   #   !"" application#   #   !"" log -> /usr/local/fooweb/shared/log#   #   !"" public#   #   !"" REVISION#   #   !"" tmp#   !"" 20110116192316#   #   !"" application#   #   !"" log -> /usr/local/fooweb/shared/log#   #   !"" public#   #   !"" REVISION#   #   !"" tmp$"" shared
  • Webistranohttps://g r/webistrano
  • Fooweb Fooweb Fooweb Mail ServiceAny SMTP Symfony 1.3 Tomcat 6.0 Server PHP 5.2.12 Java 1.6
  • !"" application#   !"" controllers#   #   $"" home.php#   $"" library#   $"" Foow#   $"" Router.php!"" fooweb.spec!"" public#   $"" index.php$"" vhosts $"" fooweb.conf
  • Summary: Fooweb ApplicationVendor: FoowebName: foowebVersion: 1.0Release: 1Source0: fooweb-%{version}.tar.gzLicense: BSDGroup: FoowebBuildArch: noarchBuildRoot: %{_tmppath}/%{name}-%{version}-buildrootRequires: php%descriptionThis is the Fooweb web application%prep%setup
  • %installmkdir -p $RPM_BUILD_ROOT/var/www/foowebmkdir -p $RPM_BUILD_ROOT/etc/httpd/conf.d/cp -r application $RPM_BUILD_ROOT/var/www/foowebcp -r public $RPM_BUILD_ROOT/var/www/foowebcp vhosts/fooweb.conf $RPM_BUILD_ROOT/etc/httpd/conf.d/%cleanrm -rf $RPM_BUILD_ROOT%files%dir /var/www%dir /var/www/fooweb%config /etc/httpd/conf.d/fooweb.conf/var/www/fooweb/*
  • ~$ mkdir buildroot buildroot/tmp~$ cat .rpmmacro%packager Fooweb Release Manager%_topdir ~/buildroot%_tmppath ~/buildroot/tmp~$ cd ~/tags~/tags$ tar cvzf fooweb-1.0.tar.gz fooweb-1.0
  • ~$ rpmbuild -ta fooweb-1.0.tar.gz~$ rpm -qip ~/rpmbuild/RPMS/noarch/fooweb-1.0-1.noarch.rpmName : foowebRelocations: (not relocatable)Version : 1.0Vendor: FoowebRelease : 1Build Date: Thu 13 Jan 2011 12:26:24 AM PSTInstall Date: (not installed)Build Host: ubuntu.localdomainGroup : FoowebSource RPM: fooweb-1.0-1.src.rpmSize : 781License : BSDSignature : (none)Summary : Fooweb ApplicationDescription : This is the Fooweb web application
  • $ mkdir /var/www/repo$ cd /var/www/repo$ mkdir centox/5/fooweb/{SRPMS,X86_64,i386,noarch}$ cp ~rpmbuild/RPMS/noarch/* centos/5/fooweb/noarch$ cp ~rpmbuild/SRPMS/* centos/5/fooweb/SRPMS$ createrepo -v centos/5/fooweb/noarch/centos/5/fooweb/noarch/!"" fooweb-1.0-1.noarch.rpm$"" repodata !"" filelists.xml.gz !"" other.xml.gz !"" primary.xml.gz $"" repomd.xml
  • $ cat /etc/yum.repos.d/fooweb.repo[fooweb_noarch]name = Fooweb Private Repositorybaseurl = = 1gpgcheck = 0gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fooweb$ yum updatefooweb_noarch 100% |========| 951 Bfooweb_noarch/primary 100% |========| 701 Bfooweb_noarch1/1Setting up Update ProcessNo Packages marked for Update
  • $ yum info foowebAvailable PackagesName : foowebArch : noarchVersion : 1.0Release : 1Size : 3.3 kRepo : fooweb_noarchSummary : Fooweb ApplicationLicense : BSDDescription: This is the Fooweb web application
  • <target name="buildrpm" depends="init"> <tar destfile="build/rpm/SOURCES/fooweb.tar.gz" compression="gzip"> <tarfileset dir="${basedir}"prefix="fooweb-1.0"> <include name="*/**" /> <exclude name="build/**" /> </tarfileset> </tar> <copy file="${basedir}/fooweb.spec" tofile="${basedir}/build/rpm/SPECS/fooweb.spec" /> <rpm command="-ba" specFile="fooweb.spec"topDir="${basedir}/build/rpm"cleanBuildDir="true" failOnError="true" /></target>
  • $ sudo aptitude install puppetmaster0 packages upgraded, 10 newly installed, 0 toremove and 76 not upgraded.Need to get 3,233kB of archives. After unpacking13.7MB will be used. Puppet$ sudo aptitude install puppet0 packages upgraded, 5 newly installed, 0 toremove and 76 not upgraded.Need to get 587kB of archives. After unpacking1,892kB will be used.
  • backend primary seconday staging Puppet master
  • /etc/puppet!"" auth.conf!"" fileserver.conf!"" manifests#   $"" site.pp!"" modules#   !"" apache2#   #   $"" manifests#   #   $"" init.pp#   $"" fooweb#   !"" files#   #   $"" fooweb.conf#   $"" manifests#   $"" init.pp!"" puppet.conf$"" templates
  • class fooweb { package { "fooweb": ensure => latest, }" file { "/etc/apache2/sites-enabled/fooweb.conf": owner => root, group => root, mode => 0444, source => "puppet:///files/fooweb/files/fooweb.conf", notify => Service["apache2"] }} mod ules/fooweb manif ests/init.pp
  • node "ubuntu.localdomain" { manifests/site.pp include fooweb include apache2}class apache2 { service { apache2: ensure => running } modules/apache2 manifests/init.pp
  • # puppet agent -o -v --no-daemonizeinfo: Caching catalog for ubuntu.localdomaininfo: Applying configuration version 1295514488notice: /Stage[main]/Fooweb/Package[fooweb]/ensure: ensure changed purged to latestnotice: /Stage[main]/Fooweb/File[/etc/apache2/sites-enabled/fooweb.conf]/ensure: definedcontent as {md5}d41d8cd98f00b204e9800998ecf8427einfo: /Stage[main]/Fooweb/File[/etc/apache2/sites-enabled/fooweb.conf]: Scheduling refresh ofService[apache2]notice: /Stage[main]/Apache2/Service[apache2]:Triggered refresh from 1 eventsnotice: Finished catalog run in 3.33 seconds# ls /etc/httpd/conf.d/fooweb.conf/etc/httpd/conf.d/fooweb.conf# ls /var/www/fooweb/application public
  • /branches/1.1.3 /trunk/tags/1.1.2 /tags/1.1.3
  • package copy code run db backup run db changes make code active
  • package copy code run db backup run db changes make code active
  • MANAGING Database ChanGES Image:
  • CREATE TABLE `blogpost` ( `id` int(11) auto_increment NOT NULL PRIMARYKEY, `title` VARCHAR(255), `timestamp` DATETIME, `content` TEXT);--//@UNDO DBDeploy tp://dbdeploy.comDROP TABLE `blogpost`; ht
  • ALTER TABLE `blogpost` ADD `author` varchar(255) NULL;--//@UNDOALTER TABLE `blogpost` DROP `author`;
  • $ wget TABLE changelog ( change_number BIGINT NOT NULL, delta_set VARCHAR(10) NOT NULL, start_dt TIMESTAMP NOT NULL, complete_dt TIMESTAMP NULL, applied_by VARCHAR(100) NOT NULL, description VARCHAR(500) NOT NULL, PRIMARY KEY(change_number, delta_set));
  • $ java -cp mysql-connector-java.jar:dbdeploy-cli-3.0M2.jar com.dbdeploy.CommandLineTarget -D com.mysql.jdbc.Driver -d mysql -o delta.sql-u jdbc:mysql://localhost/foowebdb -U root -P******dbdeploy 3.0M2Reading change scripts from directorydbdeploy.Changes currently applied to database: (none)Scripts available: 1, 2To be applied: 1, 2
  • -- START CHANGE SCRIPT #1: 1-create-blogposts.sqlCREATE TABLE `blogpost` ( `id` int(11) auto_increment NOT NULL PRIMARYKEY, `title` VARCHAR(255), `timestamp` DATETIME, `content` TEXT);INSERT INTO changelog (change_number,complete_dt, applied_by, description)VALUES(1, CURRENT_TIMESTAMP, USER(), 1-create-blogposts.sql);COMMIT;
  • -- END CHANGE SCRIPT #1: 1-create-blogposts.sql-- START CHANGE SCRIPT #2: 2-add-author.sqlALTER TABLE `blogpost` ADD `author` varchar(255)NULL;INSERT INTO changelog (change_number,complete_dt, applied_by, description) VALUES (2, CURRENT_TIMESTAMP, USER(), 2-add-author.sql);COMMIT;-- END CHANGE SCRIPT #2: 2-add-author.sql
  • <?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....]> <changeSet author="ianbarber" id="1"> <createTable tableName="blogposts"> <column autoIncrement="true" name="id"type="int(11)"> <constraints nullable="false"primaryKey="true" /> </column> <column name="title" type="varchar(255)" /> <column name="body" type="text" /> <column name="author" type="varchar(255)"/> <column name="date" type="timestamp" /> </createTable> </changeSet> Liquibase ht tp://
  • <?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....] > <include file="v000/master.xml" /></databaseChangeLog> update.xml<?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....]> <include file="v000/create-blog-posts-1.xml" /></databaseChangeLog> v000/master.xml
  • $ wget Liquibase propertiesdriver: com.mysql.jdbc.Driverurl: jdbc:mysql://localhost/foowebclasspath: /usr/share/java/mysql-connector-java.jarusername: rootpassword: ******* roperties liquibase.p
  • $ liquibase --changeLogFile=update.xml updateLiquibase Home: /opt/liquibaseINFO 1/18/11 1:32 PM:liquibase: Successfullyacquired change log lockINFO 1/18/11 1:32 PM:liquibase: Reading from`DATABASECHANGELOG`INFO 1/18/11 1:32 PM:liquibase: Reading from`DATABASECHANGELOG`INFO 1/18/11 1:32 PM:liquibase: ChangeSetv000/create-blog-posts-1.xml::1::ianbarber ransuccessfully in 101msINFO 1/18/11 1:32 PM:liquibase: Successfullyreleased change log lockLiquibase Update Successful
  • class CreateProjects < ActiveRecord::Migration def self.up create_table :projects do |t| t.column :name, :string t.column :description, :text t.column :template, :string t.column :created_at, :datetime t.column :updated_at, :datetime end end def self.down drop_table :projects endend
  • <?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....]> <changeSet author="ianbarber" id="2"> <addColumn tableName="blogposts"> <column name="commenter" type="varchar(255)" /> </addColumn> </changeSet></databaseChangeLog>$ liquibase --changeLogFile=update.xml updateLiquibase Home: /opt/liquibaseINFO 1/18/11 2:38 PM:liquibase: ChangeSet v000/add_commenter-2.xml::2::ianbarber ransuccessfully in 136msLiquibase Update Successful
  • $ liquibase --changeLogFile=update.xmlrollbackCount 1Liquibase Home: /opt/liquibaseINFO 1/18/11 2:39 PM:liquibase: Successfullyacquired change log lockINFO 1/18/11 2:39 PM:liquibase: Reading from`DATABASECHANGELOG`INFO 1/18/11 2:39 PM:liquibase: Rolling BackChangeset:v000/add_commenter-2.xml::2::ianbarber::(Checksum:3:cc45ae1014b26f8b35cb70a5fc39a1ae)INFO 1/18/11 2:39 PM:liquibase: Successfullyreleased change log lockLiquibase Rollback Successful
  • E.G. mk_ Primary slave_de http://bi lay DB DFi Replicationread read slaveslave 30 MinuteBackup Delay
  • namespace :deploy do namespace :web do task :disable, :roles => :web do on_rollback { rm "#{shared_path}/system/maintenance.html" } require erb deadline, reason = ENV[DATE], ENV[WHY] maintenance ="./templates/maintenance.erb" )).result(binding) put maintenance, "#{shared_path}/system/maintenance.html", :mode => 0644 endend
  • # DATE="16:00 MST" WHY="a database upgrade"cap deploy:web:disableif (-f $document_root/system/maintenance.html){ rewrite ^(.*)$ /system/maintenance.html last; break;}
  • warm redirectcaches & sessions links proxies migrate
  • developers devops sys QAadmins
  • See : ContinuousDeplo yment In 5Easy Stepshttp://o Image:
  • Feature F la gsImage:
  • Gradual Ramp With Feature Without Feature100%75%50%25% 0% Day 1 Day 2 Day 3 Day 4
  • Dark Lau nches
  • THanks!Deployment Tactics: Managing code from development to production Ian Barber - |