Safely         Deploying on         the cutting edge         Eric Holscher         Urban Airship         Djangocon 2011Wed...
Wednesday, September 7, 2011
Talk Contents         •   Company culture & process         •   Deployment environment         •   Tools for deploying    ...
ProcessWednesday, September 7, 2011
Process         •   Deploy out of git         •   Standard git-based production/master branch model         •   Production...
Features         •   Easily allows you to hot-fix production         •   Keep a stable master         •   Run CI on the mas...
Services         •   Everything that we deploy is conceptualized as a service         •   Services all live in /mnt/servic...
QA Environment         •   Run all of your master branches         •   Allow you to get a copy of what will become product...
Deployment Design GoalsWednesday, September 7, 2011
Jump machine         •   Have a standard place for all deployments to happen         •   Log all commands runWednesday, Se...
No External Services         •   Chishop         •   No external server required to deploy code         •   All branches a...
Services look the same         •   Python         •   Java         •   “Unix”Wednesday, September 7, 2011
Composable         •   Small pieces that you can build into better things         •   Useful when trying to do something y...
EnvironmentWednesday, September 7, 2011
Environment         •   Where code lands on the remote machine         •   Mimics a chroot         •   Uses virtualenv & s...
File Structure         •   /mnt/services/airship             •   bin/             •   current -> deploy-2011-08-03_14-37-3...
etc/         •   supervisord.conf             •   [include]             •   files = *.conf         •   airship.confWednesda...
bin/         •   start         •   stop         •   restart         •   logsWednesday, September 7, 2011
SCRIPT_DIR=$(dirname $0)         SERVICE_DIR=$(cd $SCRIPT_DIR && cd ../ && pwd)         cd $SERVICE_DIR         supervisor...
Bin scripts         •   All of the process-level binscripts wrap supervisord         •   bin/start -> supervisordctl start...
var/         •   data/         •   log/         •   run/         •   tmp/Wednesday, September 7, 2011
Init.d         •   All services share a common init.d script         •   This init.d script calls into the service’s bin/ ...
SERVICE_USER=<%= @service %>         SERVICE_NAME=<%= @service %>         SERVICE_PATH=/mnt/services/$SERVICE_NAME        ...
ToolsWednesday, September 7, 2011
Tools         •   Fabric         •   Rsync         •   Pip         •   VirtualenvWednesday, September 7, 2011
Low-level verbs         •   pull         •   build         •   tag         •   sync         •   install         •   rollba...
Pull         •   Update the code from the source repository         •   Defaults to the “production” branch             • ...
Build         •   Could be called “prepare”         •   Do local-specific things to get repo into a ready state         •  ...
Tag         •   Set a tag for the deploy in the git repo         •   If the current commit already has a tag, use that ins...
Sync         •   Move the code from the local to the remote box         •   Uses rsync to put it into the remote service d...
Install         •   Make the code the active path for code on the machine         •   This is generally installing code in...
Rollback         •   When you break things, you need to undo quickly         •   Reset the repository to the previous depl...
Start/Stop/Reload         •   Allow you to bounce services as part of deployment         •   Allow reload for services tha...
CLI UI         •   Have nice wrapper commands that do common tasks         •   deploy host:web-0 full_deploy:airship      ...
UI cont.         •   deploy host:web-0 full_deploy:airship restart:airshipWednesday, September 7, 2011
#!/bin/bash         cd ~/airdeploy         DATE=$(date +%Y_%-m_%-d-%H-%m-%s)         echo "deploy" $@ > logs/$DATE.log    ...
Meta-commands         •   Hard-code the correct deployment behavior         •   “Make easy things easy, and wrong things h...
MagicifyingWednesday, September 7, 2011
Magicifying         •   Now that we have a solid base, we can automate on top         •   When you do a meta deploy, it sh...
Workflow         •   Deploy to one web server, preferably with one worker         •   Restart it         •   Run it against...
Heuristics         •   Any 500s         •   Number of 200s to non-200s         •   Number of 500s to 200s         •   Requ...
How it works         •   Tell load balancer to take machine out of pool             •   /take_me_out_of_the_lb -> 200     ...
Thanks         •   Alex Kritikos         •   Erik Onnen         •   SchmichaelWednesday, September 7, 2011
Questions?         •   Eric Holscher         •   Urban Airship (Hiring and whatnot)         •   eric@ericholscher.comWedne...
Upcoming SlideShare
Loading in...5
×

Deploying on the cutting edge

1,818

Published on

Talk on how we do deployment at Urban Airship from Djangocon 2011.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,818
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
43
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Deploying on the cutting edge

    1. 1. Safely Deploying on the cutting edge Eric Holscher Urban Airship Djangocon 2011Wednesday, September 7, 2011
    2. 2. Wednesday, September 7, 2011
    3. 3. Talk Contents • Company culture & process • Deployment environment • Tools for deploying • Verifying deploymentWednesday, September 7, 2011
    4. 4. ProcessWednesday, September 7, 2011
    5. 5. Process • Deploy out of git • Standard git-based production/master branch model • Production branch has releases are tagged with timestamp • deploy-2011-08-03_14-37-38 • Feature branches • http://nvie.com/posts/a-successful-git-branching-model/Wednesday, September 7, 2011
    6. 6. Features • Easily allows you to hot-fix production • Keep a stable master • Run CI on the master branch or long-lived feature branchesWednesday, September 7, 2011
    7. 7. Services • Everything that we deploy is conceptualized as a service • Services all live in /mnt/services/<slug> (Thanks ec2) • A service is an instance of a repository on a machine • A repository might have multiple services • eg. Airship deployed into “celery” and “web” services • This maps really well onto Chef cookbooksWednesday, September 7, 2011
    8. 8. QA Environment • Run all of your master branches • Allow you to get a copy of what will become production • Catch errors before they are seen by customers • Spawn new ones for long-lived feature branches • `host web-0` and figure out based on IPWednesday, September 7, 2011
    9. 9. Deployment Design GoalsWednesday, September 7, 2011
    10. 10. Jump machine • Have a standard place for all deployments to happen • Log all commands runWednesday, September 7, 2011
    11. 11. No External Services • Chishop • No external server required to deploy code • All branches are checked out on an admin serverWednesday, September 7, 2011
    12. 12. Services look the same • Python • Java • “Unix”Wednesday, September 7, 2011
    13. 13. Composable • Small pieces that you can build into better things • Useful when trying to do something you didn’t plan forWednesday, September 7, 2011
    14. 14. EnvironmentWednesday, September 7, 2011
    15. 15. Environment • Where code lands on the remote machine • Mimics a chroot • Uses virtualenv & supervisord • Owned by the service-user • Managed by ChefWednesday, September 7, 2011
    16. 16. File Structure • /mnt/services/airship • bin/ • current -> deploy-2011-08-03_14-37-38 • deploy-2011-08-03_14-37-38 • etc/ • var/Wednesday, September 7, 2011
    17. 17. etc/ • supervisord.conf • [include] • files = *.conf • airship.confWednesday, September 7, 2011
    18. 18. bin/ • start • stop • restart • logsWednesday, September 7, 2011
    19. 19. SCRIPT_DIR=$(dirname $0) SERVICE_DIR=$(cd $SCRIPT_DIR && cd ../ && pwd) cd $SERVICE_DIR supervisorctl pid > /dev/null 2>&1 if [ "$?" != "0" ]; then echo "Supervisord not running, starting." supervisord else echo "Supervisord running, starting all processes." supervisorctl start all fi cd - > /dev/null 2>&1Wednesday, September 7, 2011
    20. 20. Bin scripts • All of the process-level binscripts wrap supervisord • bin/start -> supervisordctl start all • bin/start foo -> supervisorctl start foo • bin/stop -> supervisorctl stop all • bin/stop shutdown -> supervisorctl shutdownWednesday, September 7, 2011
    21. 21. var/ • data/ • log/ • run/ • tmp/Wednesday, September 7, 2011
    22. 22. Init.d • All services share a common init.d script • This init.d script calls into the service’s bin/ • /etc/init.d/airship start -> /mnt/services/airship/bin/startWednesday, September 7, 2011
    23. 23. SERVICE_USER=<%= @service %> SERVICE_NAME=<%= @service %> SERVICE_PATH=/mnt/services/$SERVICE_NAME set -e RET_CODE=0 case "$1" in start) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/start RET_CODE=$? ;; stop) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/stop RET_CODE=$? ;; restart) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/restart RET_CODE=$? ;; status) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/status RET_CODE=$? ;; *) echo "$SERVICE_NAME service usage: $0 {start|stop|restart|status}" ;; esac exit $RET_CODEWednesday, September 7, 2011
    24. 24. ToolsWednesday, September 7, 2011
    25. 25. Tools • Fabric • Rsync • Pip • VirtualenvWednesday, September 7, 2011
    26. 26. Low-level verbs • pull • build • tag • sync • install • rollback • start/stop/restart/reloadWednesday, September 7, 2011
    27. 27. Pull • Update the code from the source repository • Defaults to the “production” branch • def pull(repo=None, ref=origin/production) • Can pass in a specific revision/branch/tag/hashish • local(git reset --hard %s % ref, capture=False)Wednesday, September 7, 2011
    28. 28. Build • Could be called “prepare” • Do local-specific things to get repo into a ready state • Mostly used for compiling in java-land • Useful in Python for running pre-install tasksWednesday, September 7, 2011
    29. 29. Tag • Set a tag for the deploy in the git repo • If the current commit already has a tag, use that instead • git tag --contains HEAD • deploy-2011-08-03_14-37-38 • strftime(%Y-%m-%d_%H-%M-%S)Wednesday, September 7, 2011
    30. 30. Sync • Move the code from the local to the remote box • Uses rsync to put it into the remote service directory • Also places a copy of the synced code on the admin boxWednesday, September 7, 2011
    31. 31. Install • Make the code the active path for code on the machine • This is generally installing code into a virtualenv • Updating the “current” symlink in the service directory • Symlink Django settings file based on environmentWednesday, September 7, 2011
    32. 32. Rollback • When you break things, you need to undo quickly • Reset the repository to the previous deployed tag • git tag | grep deploy| sort -nr |head -2 |tail -1 • Deploy that • Very few moving piecesWednesday, September 7, 2011
    33. 33. Start/Stop/Reload • Allow you to bounce services as part of deployment • Allow reload for services that support itWednesday, September 7, 2011
    34. 34. CLI UI • Have nice wrapper commands that do common tasks • deploy host:web-0 full_deploy:airship ➡ pull, build, tag, sync, install • deploy host:web-1 deploy:airship ➡ tag, sync, install • deploy host:web-2 sync:airship ➡ syncWednesday, September 7, 2011
    35. 35. UI cont. • deploy host:web-0 full_deploy:airship restart:airshipWednesday, September 7, 2011
    36. 36. #!/bin/bash cd ~/airdeploy DATE=$(date +%Y_%-m_%-d-%H-%m-%s) echo "deploy" $@ > logs/$DATE.log fab $@ cd - > /dev/null 2>&1Wednesday, September 7, 2011
    37. 37. Meta-commands • Hard-code the correct deployment behavior • “Make easy things easy, and wrong things hard” • Knows what machine each service is deployed to • deploy airship ➡ deploy pull:airship ➡ deploy type:web deploy:airshipWednesday, September 7, 2011
    38. 38. MagicifyingWednesday, September 7, 2011
    39. 39. Magicifying • Now that we have a solid base, we can automate on top • When you do a meta deploy, it should be a “smart deploy”Wednesday, September 7, 2011
    40. 40. Workflow • Deploy to one web server, preferably with one worker • Restart it • Run it against heuristics to determine if it’s broken • If it’s broken, rollback, otherwise continue onWednesday, September 7, 2011
    41. 41. Heuristics • Any 500s • Number of 200s to non-200s • Number of 500s to 200s • Requests a second • Response time • $$$ (Business metrics)Wednesday, September 7, 2011
    42. 42. How it works • Tell load balancer to take machine out of pool • /take_me_out_of_the_lb -> 200 • Start your code with 1 worker and a different port • supervisorctl start canary • Expose metrics from your services over json • Make sure your load balancer weights it appropriately • Poll your metrics for X time before considering it functionalWednesday, September 7, 2011
    43. 43. Thanks • Alex Kritikos • Erik Onnen • SchmichaelWednesday, September 7, 2011
    44. 44. Questions? • Eric Holscher • Urban Airship (Hiring and whatnot) • eric@ericholscher.comWednesday, September 7, 2011
    1. ¿Le ha llamado la atención una diapositiva en particular?

      Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

    ×