Deployment Tactics

LOYM ENTDEP ICS TACT Managing code fromdevelopment to productionIan Barber - ian.barber@gmail.comtwitter.com/ianbarber | phpir.com

Image: http://ﬂickr.com/photos/denisdervisevic/4527695803

- Table of Contents -1.... Change Control2.... Environments3.... Version Control4.... The Deploy Process5.... Scripts6.... Continuous Integration7.... Remote Releases8.... Packaged Releases9.... Package Management10.. Managing Hotﬁxes11.. Managing Database Changes12.. Rollbacks13.. Tactical Deployment

Change control plan execute change changeidentify verify need close deliver

export copy to code serverrequire comparerelease md5 report restart back apache

Change Request FormRequested By: J. Teamlead Authorised By: S. ManagerSubmit Date: 2011-01-27 Change Date: 2011-02-04Reason For Change:Resolve JIRA-1602 - Listen for new .com variants on vhostChange Request:Release tag 1.1.3 via normal processmv /etc/httpd/conf.d/fooweb.conf /etc/httpd/conf.d/fooweb.oldmv ~releases/1.1.3/conf/fooweb.conf /etc/httpd/conf.d/fooweb.confVerification:http://foweb.com shows the same page as http://fooweb.comRollback:Re-release 1.1.2mv /etc/httpd/conf.d/fooweb.old /etc/httpd/conf.d/fooweb.conf

Environments Production Developmentstatic verboserobust dynamicreliable unstableoptimised experimental

on Envir onmentThe Producti Image: http://ﬂickr.com/photos/lejoe/3763218501

The Staging EnvironmentImage: http://ﬂickr.com/photos/simononly/4454401446

n Enviro nment The Int egratioImage: http://ﬂickr.com/photos/unfoldedorigami/2374016430

The Developme nt Environmen t Image: http://ﬂickr.com/photos/drewnew/511936681

VERSION CONTROL Image: http://ﬂickr.com/photos/robbie73/4346732208

/branches/newpage /branches/... /branches/search/trunk /branches/1.1.2 /tags/1.1.2

Development /branches/newpage /branches/... /branches/search/trunkIntegration /branches/1.1.2 /tags/1.1.2 Staging Production

master release1.1.1devel search feature long feature

Productionmaster release1.1.1Stagingdevel search feature Integration long feature Development

The DEPLoy PROCESS

The DEPLoy PROCESStransparent ﬂexible easy scalable graceful reliable

support SMTP process config apache vhost appupdate code config file perms packages commandslibpngupdate cache restart service service

code conﬁg packagerepository repository repository ta daserver commands deployment server server controller

code conﬁg packagerepository repository repository da taserver commands deployment server + data controller server

BUILD SCRIPTS#!/bin/bash# Deployment script for FooWeb Projectgit archive --format=tar --remote=git://repo.com/myrepo/myrepo.git HEAD -o fooweb.tartar -xf fooweb.tar /var/wwwservice httpd restart

#!/bin/bash# Deployment script for FooWeb Projectsvn export svn://localhost/fooweb-service/trunk releasecd release && mkdir buildcp -r web/* build/javac -cp /usr/share/java/servlet-api-2.5.jar -d build/WEB-INF/classes src/com/fooweb/service/*.javacd build && jar cvf ../fooweb.war * && cd ../# assumes autoDeploy is truecp fooweb.war /var/lib/tomcat6/webapps

BUILDS TOOLS tests releasecode build docs test assets results

buildtools

<?xml version="1.0" encoding="UTF-8"?><project name="FooWeb"><property name="install" location="/var/lib/tomcat6/webapps" /><property name="svn.repo" value="svn://localhost/fooweb-service/trunk" /><target name="clean"> <delete dir="build" /> <delete dir="release" /> <delete file="fooweb.war" /></target>

<target name="build"> <exec executable="svn"> <arg line="export ${svn.repo}release" /> </exec> <mkdir dir="build"/> <copy todir="build"> <fileset dir="release/web" /> </copy> <javac srcdir="release/src"destdir="build/WEB-INF/classes/"> <classpath> <pathelement path="/usr/share/java/servlet-api-2.5.jar"/> </classpath> </javac></target>

<target name="war" depends="build"> <war destfile="fooweb.war" webxml="build/WEB-INF/web.xml"> <fileset dir="build"/> <classes dir="build/WEB-INF/classes"/> </war></target><target name="deploy" depends="war"> <copy file="fooweb.war" todir="${install}" /></target></project>

$ sudo ant deployBuildfile: build.xmlbuild: [exec] Exported revision 8. [mkdir] Created dir: /tmp/build [copy] Copying 2 files to /tmp/build [copy] Copied 3 empty directories to 1 empty directory under /tmp/build [javac] Compiling 1 source file to /tmp/build/WEB-INF/classeswar: [war] Building war: /tmp/fooweb.wardeploy: [copy] Copying 1 file to /var/lib/tomcat6/webappsBUILD SUCCESSFUL Total time: 2 seconds

CONTINUOUS INTEGRATION Look at the Hud son Wiki at dson-ci.orghttp://wiki.hu

REMOTE RELEASES Image: http://ﬂickr.com/photos/scragz/309353618

server server authorized_keys authorized_keys cp ss /s h /s h ss cp deployment controller id_rsa.pubuser “deploy” ssh-keygen -t rsa

Fabricfrom fabric.api import * http://fabﬁle .org# Development environmentdef dev(): env.user = deployer env.roledefs = { "web" : [localhost], "db" : [localhost], }# Production environmentdef production(): env.user = deployer env.roledefs = { "web" : [primary.fooweb.com, secondary.fooweb.com], "db" : [backend.fooweb.com], }

# Package up release - run localdef prepare_deploy(): local(svn export svn://localhost/fooweb/trunk release) with cd(release): local(tar cvzf ../fooweb.tar.gz .) local(rm -rf release)# Restart web server@roles(web)def restart_webserver(): sudo(/etc/init.d/apache2 restart)

# Deploy to remote servers@roles(web)def deploy(): prepare_deploy() # in case of already existing with settings(warn_only=True): run(mkdir /tmp/release) run(rm -rf /tmp/release/*) put("fooweb.tar.gz", /tmp/release) with cd(/tmp/release): run("tar xvzf fooweb.tar.gz") run("rm -rf fooweb.tar.gz") run("mv * /tmp/test") restart_webserver(); local("rm -rf fooweb.tar.gz");

$ fab dev deploy[localhost] run: svn export svn://localhost/fooweb/trunk release[localhost] run: tar cvzf ../fooweb.tar.gz .[localhost] run: rm -rf release[localhost] run: mkdir /tmp/release[localhost] err: mkdir: cannot createdirectory `/tmp/release: File existsWarning: run() encountered an error (returncode 1) while executing mkdir /tmp/release[localhost] run: rm -rf /tmp/release/*[localhost] put: fooweb.tar.gz -> /tmp/release/fooweb.tar.gz

[localhost] run: tar xvzf fooweb.tar.gz[localhost] run: rm -rf fooweb.tar.gz[localhost] run: mv * /tmp/test[localhost] sudo: /etc/init.d/apache2 restartPassword for ianbarber@localhost:[localhost] out: * Restarting web serverapache2[localhost] out: ... waiting ...done.[localhost] run: rm -rf fooweb.tar.gzDone.Disconnecting from localhost... done.

$ fab production deploy[localhost] run: tar cvzf ../fooweb.tar.gz .....[primary.fooweb.com] run: rm -rf /tmp/release/[localhost] run: tar cvzf ../fooweb.tar.gz .....[secondary.fooweb.com] run: mkdir /tmp/release....Disconnecting from secondary.fooweb.com...doneDisconnecting from primary.fooweb.com... done

$ mkdir config && cd config && capify .[add] writing ./Capfile[add] making directory ./config[add] writing ./config/deploy.rb[done] capified!set :application, "set your application name "set :repository, "set your repository"set :scm, :subversionrole :web, "your web-server here"role :app, "your app-server here"role :db, "your primary db-serverhere", :primary => true Capi stranorole :db, "slave db" .com/ https:/ /github capis trano/

set :application, "fooweb"set :repository,"svn://localhost/fooweb/trunk"set :scm, :subversionset :scm_username, "deployment"set :scm_password, "s3kkr3tp4a55"set :scm_checkout, "export"set :keep_releases, 4set :normalize_asset_timestamps, falseset :deploy_to, "/usr/local/#{application}"role :web, "primary.fooweb.com"role :web, "secondary.fooweb.com"role :db, "backend.fooweb.com"

namespace :deploy do task :migrate do # nothing end task :restart do sudo "/etc/init.d/apache2 restart" endendnamespace :fooweb do task :perms do sudo "chmod -R a+w #{deploy_to}" endendafter "deploy:setup", "fooweb:perms"

$ cap deploy:setup * executing `deploy:setup * executing "sudo mkdir -p /usr/local/fooweb [...]" servers: ["primary","secondary", "backend"] [backend] executing command [...] command finished triggering after callbacks for deploy:setup * executing `fooweb:perms * executing "sudo chmod -R a+w /usr/local/fooweb" servers: ["primary","secondary","backend"] [primary] executing command [...] command finished

$ cap deploy * executing `deploy * executing `deploy:update ** transaction: start * executing `deploy:update_code executing locally: "svn info svn://localhost/fooweb/trunk -rHEAD"/usr/bin/svn * executing "svn checkout -q -r17 svn://localhost/fooweb/trunk /usr/local/fooweb/releases/20110116192456 && (echo 17 > /usr/local/fooweb/releases/20110116192456/REVISION)" servers: ["primary.fooweb.com"] [primary.fooweb.com] executing command[....] * executing `deploy:finalize_update * executing "chmod -R g+w /usr/local/fooweb/

/usr/local/fooweb/!"" current -> releases/20110116192316!"" releases# !"" 20110116190608# # !"" application# # !"" log -> /usr/local/fooweb/shared/log# # !"" public# # !"" REVISION# # !"" tmp# !"" 20110116192316# # !"" application# # !"" log -> /usr/local/fooweb/shared/log# # !"" public# # !"" REVISION# # !"" tmp$"" shared

Webistranohttps://g ithub.com/perito r/webistrano

PACKAGED RELEASESImage: http://ﬂickr.com/photos/halfbisqued/2353845688

Fooweb Fooweb Fooweb Mail ServiceAny SMTP Symfony 1.3 Tomcat 6.0 Server PHP 5.2.12 Java 1.6

!"" application# !"" controllers# # $"" home.php# $"" library# $"" Foow# $"" Router.php!"" fooweb.spec!"" public# $"" index.php$"" vhosts $"" fooweb.conf

Summary: Fooweb ApplicationVendor: FoowebName: foowebVersion: 1.0Release: 1Source0: fooweb-%{version}.tar.gzLicense: BSDGroup: FoowebBuildArch: noarchBuildRoot: %{_tmppath}/%{name}-%{version}-buildrootRequires: php%descriptionThis is the Fooweb web application%prep%setup

%installmkdir -p $RPM_BUILD_ROOT/var/www/foowebmkdir -p $RPM_BUILD_ROOT/etc/httpd/conf.d/cp -r application $RPM_BUILD_ROOT/var/www/foowebcp -r public $RPM_BUILD_ROOT/var/www/foowebcp vhosts/fooweb.conf $RPM_BUILD_ROOT/etc/httpd/conf.d/%cleanrm -rf $RPM_BUILD_ROOT%files%dir /var/www%dir /var/www/fooweb%config /etc/httpd/conf.d/fooweb.conf/var/www/fooweb/*

~$ mkdir buildroot buildroot/tmp~$ cat .rpmmacro%packager Fooweb Release Manager%_topdir ~/buildroot%_tmppath ~/buildroot/tmp~$ cd ~/tags~/tags$ tar cvzf fooweb-1.0.tar.gz fooweb-1.0

~$ rpmbuild -ta fooweb-1.0.tar.gz~$ rpm -qip ~/rpmbuild/RPMS/noarch/fooweb-1.0-1.noarch.rpmName : foowebRelocations: (not relocatable)Version : 1.0Vendor: FoowebRelease : 1Build Date: Thu 13 Jan 2011 12:26:24 AM PSTInstall Date: (not installed)Build Host: ubuntu.localdomainGroup : FoowebSource RPM: fooweb-1.0-1.src.rpmSize : 781License : BSDSignature : (none)Summary : Fooweb ApplicationDescription : This is the Fooweb web application

$ mkdir /var/www/repo$ cd /var/www/repo$ mkdir centox/5/fooweb/{SRPMS,X86_64,i386,noarch}$ cp ~rpmbuild/RPMS/noarch/* centos/5/fooweb/noarch$ cp ~rpmbuild/SRPMS/* centos/5/fooweb/SRPMS$ createrepo -v centos/5/fooweb/noarch/centos/5/fooweb/noarch/!"" fooweb-1.0-1.noarch.rpm$"" repodata !"" filelists.xml.gz !"" other.xml.gz !"" primary.xml.gz $"" repomd.xml

$ cat /etc/yum.repos.d/fooweb.repo[fooweb_noarch]name = Fooweb Private Repositorybaseurl = http://fooweb.com/repo/centos/5/fooweb/noarchenabled = 1gpgcheck = 0gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fooweb$ yum updatefooweb_noarch 100% |========| 951 Bfooweb_noarch/primary 100% |========| 701 Bfooweb_noarch1/1Setting up Update ProcessNo Packages marked for Update

$ yum info foowebAvailable PackagesName : foowebArch : noarchVersion : 1.0Release : 1Size : 3.3 kRepo : fooweb_noarchSummary : Fooweb ApplicationLicense : BSDDescription: This is the Fooweb web application

PACKAGE MANAGEMENTImage: http://ﬂickr.com/photos/southerncalifornian/2129676744

$ sudo aptitude install puppetmaster0 packages upgraded, 10 newly installed, 0 toremove and 76 not upgraded.Need to get 3,233kB of archives. After unpacking13.7MB will be used. Puppet http://puppetlabs.com$ sudo aptitude install puppet0 packages upgraded, 5 newly installed, 0 toremove and 76 not upgraded.Need to get 587kB of archives. After unpacking1,892kB will be used.

backend fooweb.com primary fooweb.com seconday fooweb.com intfooweb.com staging fooweb.com Puppet master

/etc/puppet!"" auth.conf!"" fileserver.conf!"" manifests# $"" site.pp!"" modules# !"" apache2# # $"" manifests# # $"" init.pp# $"" fooweb# !"" files# # $"" fooweb.conf# $"" manifests# $"" init.pp!"" puppet.conf$"" templates

class fooweb { package { "fooweb": ensure => latest, }" file { "/etc/apache2/sites-enabled/fooweb.conf": owner => root, group => root, mode => 0444, source => "puppet:///files/fooweb/files/fooweb.conf", notify => Service["apache2"] }} mod ules/fooweb manif ests/init.pp

node "ubuntu.localdomain" { manifests/site.pp include fooweb include apache2}class apache2 { service { apache2: ensure => running } modules/apache2 manifests/init.pp

# puppet agent -o -v --no-daemonizeinfo: Caching catalog for ubuntu.localdomaininfo: Applying configuration version 1295514488notice: /Stage[main]/Fooweb/Package[fooweb]/ensure: ensure changed purged to latestnotice: /Stage[main]/Fooweb/File[/etc/apache2/sites-enabled/fooweb.conf]/ensure: definedcontent as {md5}d41d8cd98f00b204e9800998ecf8427einfo: /Stage[main]/Fooweb/File[/etc/apache2/sites-enabled/fooweb.conf]: Scheduling refresh ofService[apache2]notice: /Stage[main]/Apache2/Service[apache2]:Triggered refresh from 1 eventsnotice: Finished catalog run in 3.33 seconds# ls /etc/httpd/conf.d/fooweb.conf/etc/httpd/conf.d/fooweb.conf# ls /var/www/fooweb/application public

MANAGING HOTFIXES Image: http://ﬂickr.com/photos/moogan/8206134

/branches/1.1.3 /trunk/tags/1.1.2 /tags/1.1.3

package copy code run db backup run db changes make code active

MANAGING Database ChanGES Image: http://ﬂickr.com/photos/theplanetdotcom/4878814847

CREATE TABLE `blogpost` ( `id` int(11) auto_increment NOT NULL PRIMARYKEY, `title` VARCHAR(255), `timestamp` DATETIME, `content` TEXT);--//@UNDO DBDeploy tp://dbdeploy.comDROP TABLE `blogpost`; ht

ALTER TABLE `blogpost` ADD `author` varchar(255) NULL;--//@UNDOALTER TABLE `blogpost` DROP `author`;

$ wget http://dbdeploy.googlecode.com/files/dbdeploy-dist-3.0M2-distribution.zipCREATE TABLE changelog ( change_number BIGINT NOT NULL, delta_set VARCHAR(10) NOT NULL, start_dt TIMESTAMP NOT NULL, complete_dt TIMESTAMP NULL, applied_by VARCHAR(100) NOT NULL, description VARCHAR(500) NOT NULL, PRIMARY KEY(change_number, delta_set));

$ java -cp mysql-connector-java.jar:dbdeploy-cli-3.0M2.jar com.dbdeploy.CommandLineTarget -D com.mysql.jdbc.Driver -d mysql -o delta.sql-u jdbc:mysql://localhost/foowebdb -U root -P******dbdeploy 3.0M2Reading change scripts from directorydbdeploy.Changes currently applied to database: (none)Scripts available: 1, 2To be applied: 1, 2

-- START CHANGE SCRIPT #1: 1-create-blogposts.sqlCREATE TABLE `blogpost` ( `id` int(11) auto_increment NOT NULL PRIMARYKEY, `title` VARCHAR(255), `timestamp` DATETIME, `content` TEXT);INSERT INTO changelog (change_number,complete_dt, applied_by, description)VALUES(1, CURRENT_TIMESTAMP, USER(), 1-create-blogposts.sql);COMMIT;

-- END CHANGE SCRIPT #1: 1-create-blogposts.sql-- START CHANGE SCRIPT #2: 2-add-author.sqlALTER TABLE `blogpost` ADD `author` varchar(255)NULL;INSERT INTO changelog (change_number,complete_dt, applied_by, description) VALUES (2, CURRENT_TIMESTAMP, USER(), 2-add-author.sql);COMMIT;-- END CHANGE SCRIPT #2: 2-add-author.sql

<?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....]> <changeSet author="ianbarber" id="1"> <createTable tableName="blogposts"> <column autoIncrement="true" name="id"type="int(11)"> <constraints nullable="false"primaryKey="true" /> </column> <column name="title" type="varchar(255)" /> <column name="body" type="text" /> <column name="author" type="varchar(255)"/> <column name="date" type="timestamp" /> </createTable> </changeSet> Liquibase ht tp://liquibase.org

<?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....] > <include file="v000/master.xml" /></databaseChangeLog> update.xml<?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....]> <include file="v000/create-blog-posts-1.xml" /></databaseChangeLog> v000/master.xml

$ wget http://downloads.sourceforge.net/project/liquibase/Liquibase%20Core/2.0.0/liquibase-2.0.0-bin.zip# Liquibase propertiesdriver: com.mysql.jdbc.Driverurl: jdbc:mysql://localhost/foowebclasspath: /usr/share/java/mysql-connector-java.jarusername: rootpassword: ******* roperties liquibase.p

$ liquibase --changeLogFile=update.xml updateLiquibase Home: /opt/liquibaseINFO 1/18/11 1:32 PM:liquibase: Successfullyacquired change log lockINFO 1/18/11 1:32 PM:liquibase: Reading from`DATABASECHANGELOG`INFO 1/18/11 1:32 PM:liquibase: Reading from`DATABASECHANGELOG`INFO 1/18/11 1:32 PM:liquibase: ChangeSetv000/create-blog-posts-1.xml::1::ianbarber ransuccessfully in 101msINFO 1/18/11 1:32 PM:liquibase: Successfullyreleased change log lockLiquibase Update Successful

class CreateProjects < ActiveRecord::Migration def self.up create_table :projects do |t| t.column :name, :string t.column :description, :text t.column :template, :string t.column :created_at, :datetime t.column :updated_at, :datetime end end def self.down drop_table :projects endend

ROLLING BACK Image: http://ﬂickr.com/photos/roolrool/4758613588

<?xml version="1.0" encoding="UTF-8"standalone="no"?><databaseChangeLog [....]> <changeSet author="ianbarber" id="2"> <addColumn tableName="blogposts"> <column name="commenter" type="varchar(255)" /> </addColumn> </changeSet></databaseChangeLog>$ liquibase --changeLogFile=update.xml updateLiquibase Home: /opt/liquibaseINFO 1/18/11 2:38 PM:liquibase: ChangeSet v000/add_commenter-2.xml::2::ianbarber ransuccessfully in 136msLiquibase Update Successful

$ liquibase --changeLogFile=update.xmlrollbackCount 1Liquibase Home: /opt/liquibaseINFO 1/18/11 2:39 PM:liquibase: Successfullyacquired change log lockINFO 1/18/11 2:39 PM:liquibase: Reading from`DATABASECHANGELOG`INFO 1/18/11 2:39 PM:liquibase: Rolling BackChangeset:v000/add_commenter-2.xml::2::ianbarber::(Checksum:3:cc45ae1014b26f8b35cb70a5fc39a1ae)INFO 1/18/11 2:39 PM:liquibase: Successfullyreleased change log lockLiquibase Rollback Successful

TACTICALDEPLOYMENTS Image: http://ﬂickr.com/photos/romainguy/230416692

E.G. mk_ Primary slave_de http://bi lay DB t.ly/hDW DFi Replicationread read slaveslave 30 MinuteBackup Delay

namespace :deploy do namespace :web do task :disable, :roles => :web do on_rollback { rm "#{shared_path}/system/maintenance.html" } require erb deadline, reason = ENV[DATE], ENV[WHY] maintenance = ERB.new( File.read("./templates/maintenance.erb" )).result(binding) put maintenance, "#{shared_path}/system/maintenance.html", :mode => 0644 endend

# DATE="16:00 MST" WHY="a database upgrade"cap deploy:web:disableif (-f $document_root/system/maintenance.html){ rewrite ^(.*)$ /system/maintenance.html last; break;}

warm redirectcaches & sessions links proxies migrate

developers devops sys QAadmins

See : ContinuousDeplo yment In 5Easy Stepshttp://o reil.ly/13EPgd Image: http://ﬂickr.com/photos/jurvetson/3961794276

Feature F la gsImage: http://ﬂickr.com/photos/rossharmes/4153769740

Gradual Ramp With Feature Without Feature100%75%50%25% 0% Day 1 Day 2 Day 3 Day 4

Dark Lau nches

THanks!Deployment Tactics: Managing code from development to production Ian Barber - ian.barber@gmail.com twitter.com/ianbarber | phpir.com

Deployment Tactics

by Ian Barber on Jan 27, 2011

Accessibility

Tags

Upload Details

Usage Rights

Statistics

Deployment Tactics — Presentation Transcript