Python Deployment with Fabric


Published on

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Launched 3 sites very quickly. Spent some time working on platform, making it very robust. Tools. Going back to site mode soon.
  • Almost everyone starts out with Django writing a Blog.
  • Webserver: Use Apache, mod_wsgi, daemon mode. Threads more efficient, but make sure your code is thread safe. If in doubt, use processes.
  • put nginx in front of apache. nginx serves media directory. forwards app requests to apache. load balancing becomes easy
  • Probably need data storage, probably a relational DB: PostgreSQL or MySQL For tips on tuning, visit Frank Wiles @revsys for PostgreSQL or Percona ( for MySQL.
  • Do yourself a favor and run Memcached. Having a small server, even as little as 16MB will help tremendously with Django's Cache middleware.
  • With a single server, you're able to get away with simple deployment. Manual copy of files via SCP or Rsync. Manual restart the web server or touch the wsgi file. Manual DB migrations.
  • Moving beyond a single server. Move the database to it's own box. Lots of RAM. As much as you can afford. If you do a lot of writes, get fast disks, too. SCSI 15k RPM. Take advantage of RAID if you can afford it. Tune your DB setup.
  • Add more web servers. Your app will either be CPU or memory bound. Characteristics different per app. Use a tool like Munin to visualize and understand your app's resource usage.
  • All sorts of additional boxes that need some or all of your code base to operate.
  • Basic deployment isn't so basic anymore.
  • More deployment requirements - deploy media to S3/Cloudfront or your favorite CDN - splash page to indicate site maintenance - db migrations - add a new web server to the cluster - install/upgrade python package dependencies - execution of arbitrary commands (invoke) for easy sys-admin tasks
  • Two options... Capistrano or Fabric Capistrano more mature. + Out of the box support for a number of common tasks. + Before / After hooks built in for each task to allow for extending. + Interfaces for all common source control mgmt systems + Supports simultaneous deployment to all hosts at once. - It's in Ruby
  • Fabric is rather new, not nearly as mature. Doesn't have a lot of stuff built in besides command invocation. Everyone's deployment needs vary because everyone has slightly different production architectures. Not the end of the world that Fabric doesn't have everything built in.
  • Core functions: local() run() sudo()
  • Fabric's auth model is based on the underlying SSH model.
  • Fabric stores config info in it's environment object. just a dictionary, can add any info you want
  • Use env.roledefs dictionary to configure roles-to-hosts. Since it's just a dictionary, you can store whatever config data you want
  • The nuts and bolts of the command execution model. Essentially you invoke one or more tasks. Tasks can in turn invoke other tasks. Tasks are just Python functions. Any function within that doesn't begin with an _ is considered a task. TIP: If you're importing functions into your for use, import the module they're contained in instead. That way they don't show up as tasks.
  • Decorators
  • There's some problems with Fabric's role system out of the box. Attempting to override the defaults from the command line don't really work well. The maintainers acknowledge this -- and they're waiting on a fix until they nail down the syntax. But they're easy enough to fix now by writing our own decorators. These decorators extend Fabric's built in @roles and @hosts. With these, If you don't specify a host or role on the command line, the tasks will be executed with whatever the default is provided in the decorator. This is how the docs suggest Fabric is supposed to work -- but I didn't have luck with it.
  • Once we have a sane execution model, we can continue writing some tasks.
  • List of Whiskey's Fabric tasks
  • Chaining of tasks is as simple as one Python function calling another. Each function being referenced here is a separate Fabric task, each of which could be called independently. 95% of the time, this function is used.
  • Python Deployment with Fabric

    1. 1. Prodution Architecture and Deployment with Fabric - Andy McCurdy - @andymccurdy
    2. 2. Whiskey Media
    3. 3. Whiskey Sites
    4. 4. Your First Django App
    5. 5. Basic Config (web) <ul><ul><li>Apache </li></ul></ul><ul><ul><li>mod_wsgi </li></ul></ul><ul><ul><ul><li>use daemon mode </li></ul></ul></ul><ul><ul><ul><li>threads more efficient </li></ul></ul></ul><ul><ul><ul><li>processes if you're unsure of thread safety </li></ul></ul></ul><ul><li>WSGIDaemonProcess my-site python-path=/home/code/ </li></ul><ul><li>processes=2 threads=150 maximum-requests=5000 </li></ul><ul><li>WSGIProcessGroup my-site </li></ul><ul><li>WSGIScriptAlias / /home/code/my-site/deploy/wsgi/my-site.wsgi </li></ul>
    6. 6. Basic Config (media) <ul><ul><li>Nginx </li></ul></ul><ul><ul><ul><li>Use Nginx to proxy traffic to Apache </li></ul></ul></ul><ul><ul><ul><li>Meanwhile Nginx serves media </li></ul></ul></ul><ul><li>upstream my-site { </li></ul><ul><li>server; </li></ul><ul><li>} </li></ul><ul><li>server { </li></ul><ul><li>listen 80; </li></ul><ul><li>location ~ ^/media/ { </li></ul><ul><li>root /home/code/my-site; </li></ul><ul><li>expires 30d; </li></ul><ul><li>} </li></ul><ul><li>location / { </li></ul><ul><li>proxy_pass http://my-site; </li></ul><ul><li>proxy_set_header X-Real-IP $remote_addr; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
    7. 7. Basic Config (db) <ul><ul><li>Databases </li></ul></ul><ul><ul><ul><li>PostgreSQL </li></ul></ul></ul><ul><ul><ul><ul><li>Frank Wiles @ </li></ul></ul></ul></ul><ul><ul><ul><li>MySQL </li></ul></ul></ul><ul><ul><ul><ul><li>Percona @ </li></ul></ul></ul></ul>
    8. 8. Basic Config (cache) <ul><ul><li>Use Memcached! Even 16mb will significantly help against a Digg or being Slashdot'ed </li></ul></ul><ul><ul><li>It's incredibly easy... </li></ul></ul><ul><li># </li></ul><ul><li>MIDDLEWARE_CLASSES = ( </li></ul><ul><li>'django.middleware.cache.UpdateCacheMiddleware', </li></ul><ul><li>'django.middleware.common.CommonMiddleware', </li></ul><ul><li>'django.middleware.cache.FetchFromCacheMiddleware') </li></ul><ul><li>CACHE_BACKEND = 'memcached://' </li></ul><ul><li>CACHE_MIDDLEWARE_SECONDS = 60*5 # 5 minutes </li></ul><ul><li>CACHE_MIDDLEWARE_ANONYMOUS_ONLY = True </li></ul>
    9. 9. Basic Deployment <ul><ul><li>Copy Code & Media (rsync or scp) </li></ul></ul><ul><ul><li>Run DB migrations / syncdb </li></ul></ul><ul><ul><li>Bounce wsgi daemons </li></ul></ul>
    10. 10. Managing Growth (db) <ul><ul><li>Move DB to its own server </li></ul></ul><ul><ul><ul><li>As much RAM as possible </li></ul></ul></ul><ul><ul><ul><li>Write-heavy (>10%)? Get fast disks </li></ul></ul></ul><ul><ul><ul><li>Tune your config file </li></ul></ul></ul>
    11. 11. Managing Growth (web) <ul><ul><li>Add more web servers </li></ul></ul><ul><ul><li>Use a resource monitoring tool like Munin to understand if your app is CPU or memory bound </li></ul></ul>
    12. 12. Even More Growth <ul><ul><li>Replicated or sharded Databases </li></ul></ul><ul><ul><li>Multiple load balancers for redundancy </li></ul></ul><ul><ul><li>Message queues </li></ul></ul><ul><ul><li>Crons </li></ul></ul><ul><ul><li>Search daemons (Solr, Sphinx) </li></ul></ul><ul><ul><li>etc... </li></ul></ul>
    13. 13. (not so) Basic Deployment image credit: Brad Fitzpatrick
    14. 14. Deployment Reqs <ul><ul><li>Copy media to CDN </li></ul></ul><ul><ul><li>Maintenance splash page </li></ul></ul><ul><ul><li>Run DB migrations </li></ul></ul><ul><ul><li>Install/Upgrade Python dependencies </li></ul></ul><ul><ul><li>Add a new web server to the cluster </li></ul></ul><ul><ul><li>Execute arbritrary commands for sysadmin maintenance tasks </li></ul></ul>
    15. 15. Deployment Options <ul><li>Capistrano </li></ul><ul><li>+ Out of box support for common use cases </li></ul><ul><li>+ Hooks to customize tasks </li></ul><ul><li>+ Source control integration </li></ul><ul><li>+ Threaded deployment to multiple hosts </li></ul><ul><li>- Ruby :( </li></ul>
    16. 16. Deployment Options <ul><li>Fabric </li></ul><ul><li>+ Very simple, tasks are just Python functions </li></ul><ul><li>+ Easy to chain together tasks to create complex scripts out of bite size pieces </li></ul><ul><li>- No source control integration </li></ul><ul><li>- No out of box support </li></ul><ul><li>- Some bugs, although fairly easy to work around, and new maintainer is working on fixes </li></ul>
    17. 17. Fabric Basics <ul><ul><li>sudo easy_install fabric </li></ul></ul><ul><ul><li>need a </li></ul></ul><ul><ul><li>from fabric.api import * </li></ul></ul><ul><ul><li>be mindful of tasks that may fail </li></ul></ul><ul><ul><li>each remote command starts fresh </li></ul></ul><ul><ul><ul><li>changing directories </li></ul></ul></ul>
    18. 18. Core Functionality <ul><ul><li>local() - Run a command locally </li></ul></ul><ul><ul><li>run() - Run a command remotely </li></ul></ul><ul><ul><li>sudo() - Run a command remotely as another user </li></ul></ul><ul><ul><li>put() - Copy a file from local to remote </li></ul></ul><ul><ul><li>get() - Copy a file from remote to local </li></ul></ul><ul><ul><li>many more helper-ish commands </li></ul></ul>
    19. 19. Authentication <ul><ul><li>Relies on SSH model </li></ul></ul><ul><ul><li>Use SSH keys </li></ul></ul><ul><ul><li>Control access to root user via sudoers </li></ul></ul><ul><ul><li>When you have to revoke access, you just turn off their SSH account </li></ul></ul>
    20. 20. Configuration <ul><ul><li>Fabric environment (env) -- it's just a dictionary </li></ul></ul><ul><ul><ul><li>Hosts and Roles </li></ul></ul></ul><ul><ul><ul><li>Code Repositories </li></ul></ul></ul><ul><ul><ul><li>Whatever you need </li></ul></ul></ul><ul><ul><li>~/fabricrc </li></ul></ul><ul><ul><ul><li>Global settings or all Fabric deployments </li></ul></ul></ul><ul><ul><ul><li>SSH username </li></ul></ul></ul>
    21. 21. Example Config <ul><li># </li></ul><ul><li>from fabric.api import * </li></ul><ul><li>env.roledefs = { </li></ul><ul><li>'web' : ['', ''], </li></ul><ul><li>'db' : [''], </li></ul><ul><li>'lb' : [''], </li></ul><ul><li>} </li></ul><ul><li>env.repositories = {...} </li></ul>
    22. 22. Tasks <ul><li># </li></ul><ul><li>def uptime(): </li></ul><ul><li>run('uptime') </li></ul><ul><li>$> fab uptime -H </li></ul><ul><li>[] run: uptime </li></ul><ul><li>[] out: 05:20:39 up 88 days, 12:00, 0 users, load average: 0.03, 0.03, 0.00 </li></ul>
    23. 23. Mapping Roles to Tasks <ul><li># </li></ul><ul><li>@roles('web') </li></ul><ul><li>def uptime(): </li></ul><ul><li>run('uptime') </li></ul><ul><li>$> fab uptime </li></ul><ul><li>[] run: uptime </li></ul><ul><li>[] out: 05:20:39 up 88 days... </li></ul><ul><li>[] run: uptime </li></ul><ul><li>[] out: 05:20:39 up 88 days... </li></ul>
    24. 24. Decorator Problems <ul><ul><li>Some problems with Fabric's role management </li></ul></ul><ul><ul><li>Can't override decorated tasks at command line as docs suggest </li></ul></ul><ul><li>def default_roles(*role_list): </li></ul><ul><li>def selectively_attach(func): </li></ul><ul><li>if not env.roles and not env.hosts: </li></ul><ul><li>return roles(*role_list)(func) </li></ul><ul><li>else: </li></ul><ul><li>if env.hosts: </li></ul><ul><li>func = hosts(*env.hosts)(func) </li></ul><ul><li>if env.roles: </li></ul><ul><li>func = roles(*env.roles)(func) </li></ul><ul><li>return func </li></ul><ul><li>return selectively_attach </li></ul>
    25. 25. All better now <ul><li> </li></ul><ul><li>@default_roles('web', 'db') </li></ul><ul><li>def uptime(): </li></ul><ul><li>run('uptime') </li></ul><ul><li>$> fab uptime </li></ul><ul><li># runs on all hosts in the 'web' and 'db' roles </li></ul><ul><li>$> fab uptime --roles lb </li></ul><ul><li># runs only on hosts in the 'lb' role </li></ul>
    26. 26. Dealing with Failures <ul><ul><li>By default Fabric dies if a task fails </li></ul></ul><ul><ul><li>Use a context manager when failures are anticipated </li></ul></ul><ul><li># </li></ul><ul><li>from __future__ import with_statement # py2.5 </li></ul><ul><li>def symlink_me(): </li></ul><ul><li>with settings(warn_only=True): </li></ul><ul><li>run('rm /path/to/symlink') </li></ul><ul><li>run('ln -s /home/andy /path/to/symlink') </li></ul>
    27. 27. Easy sys-admin <ul><ul><li>Make an &quot;invoke&quot; command </li></ul></ul><ul><ul><li>Great for sys-admin and one-off tasks </li></ul></ul><ul><li># </li></ul><ul><li>@default_roles('all') </li></ul><ul><li>def invoke(command): </li></ul><ul><li>&quot;Invoke an arbritrary command&quot; </li></ul><ul><li>sudo(command) </li></ul><ul><li># install new packages on all hosts in one command </li></ul><ul><li>$> fab invoke:&quot;apt-get install git-core&quot; </li></ul>
    28. 28. Real World Tasks <ul><li>$> fab --list </li></ul><ul><li>Available commands: </li></ul><ul><li>bounce_wsgi_procs Bounce the WSGI procs by touching the files </li></ul><ul><li>deploy Full deployment </li></ul><ul><li>deploy_media Push media to S3 </li></ul><ul><li>invoke Invoke an arbritrary command </li></ul><ul><li>migrate Run any migrations via South </li></ul><ul><li>reload_nginx Update Nginx's running config </li></ul><ul><li>splash_off Configure Nginx to serve the site </li></ul><ul><li>splash_on Configure Nginx to serve a downed-site page </li></ul><ul><li>update_repositories Push code to servers </li></ul><ul><li>update_dependencies Update dependencies to third party libs </li></ul>
    29. 29. Whiskey's Deployment <ul><li>def deploy(splash='no'): </li></ul><ul><li>&quot;Full deployment&quot; </li></ul><ul><li>deploy_media() </li></ul><ul><li>update_cached_repositories() </li></ul><ul><li>update_dependencies() </li></ul><ul><li>generate_releases() </li></ul><ul><li>if splash == 'yes': </li></ul><ul><li>splash_on() </li></ul><ul><li>_symlink_code() </li></ul><ul><li>migrate() </li></ul><ul><li>bounce_wsgi_procs() </li></ul><ul><li>if splash == 'yes': </li></ul><ul><li>splash_off() </li></ul><ul><li>$> fab deploy:splash=yes </li></ul>
    30. 30. <ul><li>Questions? </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.