Deploying on the cutting edge

Safely
Deploying on
the cutting edge
Eric Holscher
Urban Airship
Djangocon 2011

Wednesday, September 7, 2011

Talk Contents

• Company culture & process
• Deployment environment
• Tools for deploying
• Verifying deployment


Process


Process

• Deploy out of git
• Standard git-based production/master branch model
• Production branch has releases are tagged with timestamp
• deploy-2011-08-03_14-37-38
• Feature branches
• http://nvie.com/posts/a-successful-git-branching-model/


Features

• Easily allows you to hot-ﬁx production
• Keep a stable master
• Run CI on the master branch or long-lived feature branches


Services

• Everything that we deploy is conceptualized as a service
• Services all live in /mnt/services/<slug> (Thanks ec2)
• A service is an instance of a repository on a machine
• A repository might have multiple services
• eg. Airship deployed into “celery” and “web” services
• This maps really well onto Chef cookbooks


QA Environment

• Run all of your master branches
• Allow you to get a copy of what will become production
• Catch errors before they are seen by customers
• Spawn new ones for long-lived feature branches
• `host web-0` and ﬁgure out based on IP


Deployment Design Goals


Jump machine

• Have a standard place for all deployments to happen
• Log all commands run


No External Services

• Chishop
• No external server required to deploy code
• All branches are checked out on an admin server


Services look the same

• Python
• Java
• “Unix”


Composable

• Small pieces that you can build into better things
• Useful when trying to do something you didn’t plan for


Environment


Environment

• Where code lands on the remote machine
• Mimics a chroot
• Uses virtualenv & supervisord
• Owned by the service-user
• Managed by Chef


File Structure

• /mnt/services/airship
• bin/
• current -> deploy-2011-08-03_14-37-38
• deploy-2011-08-03_14-37-38
• etc/
• var/


etc/

• supervisord.conf
• [include]
• ﬁles = *.conf
• airship.conf


bin/

• start
• stop
• restart
• logs


SCRIPT_DIR=$(dirname $0)
SERVICE_DIR=$(cd $SCRIPT_DIR && cd ../ && pwd)

cd $SERVICE_DIR
supervisorctl pid > /dev/null 2>&1
if [ "$?" != "0" ]; then
echo "Supervisord not running, starting."
supervisord
else
echo "Supervisord running, starting all processes."
supervisorctl start all
fi
cd - > /dev/null 2>&1


Bin scripts

• All of the process-level binscripts wrap supervisord
• bin/start -> supervisordctl start all
• bin/start foo -> supervisorctl start foo
• bin/stop -> supervisorctl stop all
• bin/stop shutdown -> supervisorctl shutdown


var/

• data/
• log/
• run/
• tmp/


Init.d

• All services share a common init.d script
• This init.d script calls into the service’s bin/
• /etc/init.d/airship start -> /mnt/services/airship/bin/start


SERVICE_USER='<%= @service %>'
SERVICE_NAME='<%= @service %>'
SERVICE_PATH=/mnt/services/$SERVICE_NAME
set -e
RET_CODE=0
case "$1" in
start)
sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/start
RET_CODE=$?
;;
stop)
sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/stop
RET_CODE=$?
;;
restart)
sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/restart
RET_CODE=$?
;;
status)
sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/status
RET_CODE=$?
;;
*)
echo "$SERVICE_NAME service usage: $0 {start|stop|restart|status}"
;;
esac

exit $RET_CODE


Tools


Tools

• Fabric
• Rsync
• Pip
• Virtualenv


Low-level verbs

• pull
• build
• tag
• sync
• install
• rollback
• start/stop/restart/reload


Pull

• Update the code from the source repository
• Defaults to the “production” branch
• def pull(repo=None, ref='origin/production')
• Can pass in a speciﬁc revision/branch/tag/hashish
• local('git reset --hard %s' % ref, capture=False)


Build

• Could be called “prepare”
• Do local-speciﬁc things to get repo into a ready state
• Mostly used for compiling in java-land
• Useful in Python for running pre-install tasks


Tag

• Set a tag for the deploy in the git repo
• If the current commit already has a tag, use that instead
• git tag --contains HEAD
• deploy-2011-08-03_14-37-38
• strftime('%Y-%m-%d_%H-%M-%S')


Sync

• Move the code from the local to the remote box
• Uses rsync to put it into the remote service directory
• Also places a copy of the synced code on the admin box


Install

• Make the code the active path for code on the machine
• This is generally installing code into a virtualenv
• Updating the “current” symlink in the service directory
• Symlink Django settings ﬁle based on environment


Rollback

• When you break things, you need to undo quickly
• Reset the repository to the previous deployed tag
• git tag | grep deploy| sort -nr |head -2 |tail -1
• Deploy that
• Very few moving pieces


Start/Stop/Reload

• Allow you to bounce services as part of deployment
• Allow reload for services that support it


CLI UI

• Have nice wrapper commands that do common tasks
• deploy host:web-0 full_deploy:airship
➡ pull, build, tag, sync, install
• deploy host:web-1 deploy:airship
➡ tag, sync, install
• deploy host:web-2 sync:airship
➡ sync


UI cont.

• deploy host:web-0 full_deploy:airship restart:airship


#!/bin/bash

cd ~/airdeploy
DATE=$(date +%Y_%-m_%-d-%H-%m-%s)
echo "deploy" $@ > logs/$DATE.log
fab $@
cd - > /dev/null 2>&1


Meta-commands

• Hard-code the correct deployment behavior
• “Make easy things easy, and wrong things hard”
• Knows what machine each service is deployed to
• deploy airship
➡ deploy pull:airship
➡ deploy type:web deploy:airship


Magicifying


Magicifying

• Now that we have a solid base, we can automate on top
• When you do a meta deploy, it should be a “smart deploy”


Workﬂow

• Deploy to one web server, preferably with one worker
• Restart it
• Run it against heuristics to determine if it’s broken
• If it’s broken, rollback, otherwise continue on


Heuristics

• Any 500s
• Number of 200s to non-200s
• Number of 500s to 200s
• Requests a second
• Response time
• $$$ (Business metrics)


How it works

• Tell load balancer to take machine out of pool
• /take_me_out_of_the_lb -> 200
• Start your code with 1 worker and a different port
• supervisorctl start canary
• Expose metrics from your services over json
• Make sure your load balancer weights it appropriately
• Poll your metrics for X time before considering it functional


Thanks

• Alex Kritikos
• Erik Onnen
• Schmichael


Questions?

• Eric Holscher
• Urban Airship (Hiring and whatnot)
• eric@ericholscher.com


Deploying on the cutting edge

More Related Content

What's hot

Similar to Deploying on the cutting edge

More from ericholscher

Recently uploaded

Deploying on the cutting edge

Editor's Notes