Deploying on the cutting edge
Upcoming SlideShare
Loading in...5
×
 

Deploying on the cutting edge

on

  • 2,031 views

Talk on how we do deployment at Urban Airship from Djangocon 2011.

Talk on how we do deployment at Urban Airship from Djangocon 2011.

Statistics

Views

Total Views
2,031
Views on SlideShare
1,856
Embed Views
175

Actions

Likes
5
Downloads
43
Comments
0

5 Embeds 175

http://lanyrd.com 168
http://paper.li 4
http://twitter.com 1
http://tweetedtimes.com 1
https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Deploying on the cutting edge Deploying on the cutting edge Presentation Transcript

  • Safely Deploying on the cutting edge Eric Holscher Urban Airship Djangocon 2011Wednesday, September 7, 2011
  • Wednesday, September 7, 2011
  • Talk Contents • Company culture & process • Deployment environment • Tools for deploying • Verifying deploymentWednesday, September 7, 2011
  • ProcessWednesday, September 7, 2011
  • Process • Deploy out of git • Standard git-based production/master branch model • Production branch has releases are tagged with timestamp • deploy-2011-08-03_14-37-38 • Feature branches • http://nvie.com/posts/a-successful-git-branching-model/Wednesday, September 7, 2011
  • Features • Easily allows you to hot-fix production • Keep a stable master • Run CI on the master branch or long-lived feature branchesWednesday, September 7, 2011
  • Services • Everything that we deploy is conceptualized as a service • Services all live in /mnt/services/<slug> (Thanks ec2) • A service is an instance of a repository on a machine • A repository might have multiple services • eg. Airship deployed into “celery” and “web” services • This maps really well onto Chef cookbooksWednesday, September 7, 2011
  • QA Environment • Run all of your master branches • Allow you to get a copy of what will become production • Catch errors before they are seen by customers • Spawn new ones for long-lived feature branches • `host web-0` and figure out based on IPWednesday, September 7, 2011
  • Deployment Design GoalsWednesday, September 7, 2011
  • Jump machine • Have a standard place for all deployments to happen • Log all commands runWednesday, September 7, 2011
  • No External Services • Chishop • No external server required to deploy code • All branches are checked out on an admin serverWednesday, September 7, 2011
  • Services look the same • Python • Java • “Unix”Wednesday, September 7, 2011
  • Composable • Small pieces that you can build into better things • Useful when trying to do something you didn’t plan forWednesday, September 7, 2011
  • EnvironmentWednesday, September 7, 2011
  • Environment • Where code lands on the remote machine • Mimics a chroot • Uses virtualenv & supervisord • Owned by the service-user • Managed by ChefWednesday, September 7, 2011
  • File Structure • /mnt/services/airship • bin/ • current -> deploy-2011-08-03_14-37-38 • deploy-2011-08-03_14-37-38 • etc/ • var/Wednesday, September 7, 2011
  • etc/ • supervisord.conf • [include] • files = *.conf • airship.confWednesday, September 7, 2011
  • bin/ • start • stop • restart • logsWednesday, September 7, 2011
  • SCRIPT_DIR=$(dirname $0) SERVICE_DIR=$(cd $SCRIPT_DIR && cd ../ && pwd) cd $SERVICE_DIR supervisorctl pid > /dev/null 2>&1 if [ "$?" != "0" ]; then echo "Supervisord not running, starting." supervisord else echo "Supervisord running, starting all processes." supervisorctl start all fi cd - > /dev/null 2>&1Wednesday, September 7, 2011
  • Bin scripts • All of the process-level binscripts wrap supervisord • bin/start -> supervisordctl start all • bin/start foo -> supervisorctl start foo • bin/stop -> supervisorctl stop all • bin/stop shutdown -> supervisorctl shutdownWednesday, September 7, 2011
  • var/ • data/ • log/ • run/ • tmp/Wednesday, September 7, 2011
  • Init.d • All services share a common init.d script • This init.d script calls into the service’s bin/ • /etc/init.d/airship start -> /mnt/services/airship/bin/startWednesday, September 7, 2011
  • SERVICE_USER=<%= @service %> SERVICE_NAME=<%= @service %> SERVICE_PATH=/mnt/services/$SERVICE_NAME set -e RET_CODE=0 case "$1" in start) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/start RET_CODE=$? ;; stop) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/stop RET_CODE=$? ;; restart) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/restart RET_CODE=$? ;; status) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/status RET_CODE=$? ;; *) echo "$SERVICE_NAME service usage: $0 {start|stop|restart|status}" ;; esac exit $RET_CODEWednesday, September 7, 2011
  • ToolsWednesday, September 7, 2011
  • Tools • Fabric • Rsync • Pip • VirtualenvWednesday, September 7, 2011
  • Low-level verbs • pull • build • tag • sync • install • rollback • start/stop/restart/reloadWednesday, September 7, 2011
  • Pull • Update the code from the source repository • Defaults to the “production” branch • def pull(repo=None, ref=origin/production) • Can pass in a specific revision/branch/tag/hashish • local(git reset --hard %s % ref, capture=False)Wednesday, September 7, 2011
  • Build • Could be called “prepare” • Do local-specific things to get repo into a ready state • Mostly used for compiling in java-land • Useful in Python for running pre-install tasksWednesday, September 7, 2011
  • Tag • Set a tag for the deploy in the git repo • If the current commit already has a tag, use that instead • git tag --contains HEAD • deploy-2011-08-03_14-37-38 • strftime(%Y-%m-%d_%H-%M-%S)Wednesday, September 7, 2011
  • Sync • Move the code from the local to the remote box • Uses rsync to put it into the remote service directory • Also places a copy of the synced code on the admin boxWednesday, September 7, 2011
  • Install • Make the code the active path for code on the machine • This is generally installing code into a virtualenv • Updating the “current” symlink in the service directory • Symlink Django settings file based on environmentWednesday, September 7, 2011
  • Rollback • When you break things, you need to undo quickly • Reset the repository to the previous deployed tag • git tag | grep deploy| sort -nr |head -2 |tail -1 • Deploy that • Very few moving piecesWednesday, September 7, 2011
  • Start/Stop/Reload • Allow you to bounce services as part of deployment • Allow reload for services that support itWednesday, September 7, 2011
  • CLI UI • Have nice wrapper commands that do common tasks • deploy host:web-0 full_deploy:airship ➡ pull, build, tag, sync, install • deploy host:web-1 deploy:airship ➡ tag, sync, install • deploy host:web-2 sync:airship ➡ syncWednesday, September 7, 2011
  • UI cont. • deploy host:web-0 full_deploy:airship restart:airshipWednesday, September 7, 2011
  • #!/bin/bash cd ~/airdeploy DATE=$(date +%Y_%-m_%-d-%H-%m-%s) echo "deploy" $@ > logs/$DATE.log fab $@ cd - > /dev/null 2>&1Wednesday, September 7, 2011
  • Meta-commands • Hard-code the correct deployment behavior • “Make easy things easy, and wrong things hard” • Knows what machine each service is deployed to • deploy airship ➡ deploy pull:airship ➡ deploy type:web deploy:airshipWednesday, September 7, 2011
  • MagicifyingWednesday, September 7, 2011
  • Magicifying • Now that we have a solid base, we can automate on top • When you do a meta deploy, it should be a “smart deploy”Wednesday, September 7, 2011
  • Workflow • Deploy to one web server, preferably with one worker • Restart it • Run it against heuristics to determine if it’s broken • If it’s broken, rollback, otherwise continue onWednesday, September 7, 2011
  • Heuristics • Any 500s • Number of 200s to non-200s • Number of 500s to 200s • Requests a second • Response time • $$$ (Business metrics)Wednesday, September 7, 2011
  • How it works • Tell load balancer to take machine out of pool • /take_me_out_of_the_lb -> 200 • Start your code with 1 worker and a different port • supervisorctl start canary • Expose metrics from your services over json • Make sure your load balancer weights it appropriately • Poll your metrics for X time before considering it functionalWednesday, September 7, 2011
  • Thanks • Alex Kritikos • Erik Onnen • SchmichaelWednesday, September 7, 2011
  • Questions? • Eric Holscher • Urban Airship (Hiring and whatnot) • eric@ericholscher.comWednesday, September 7, 2011