Best Of Both Worlds
Upcoming SlideShare
Loading in...5
×
 

Best Of Both Worlds

on

  • 3,018 views

While Django is a great Web framework, there are places where Python ecosystem falls short of its competitors. However, nothing prevents us from picking the best parts to create a mixed environment ...

While Django is a great Web framework, there are places where Python ecosystem falls short of its competitors. However, nothing prevents us from picking the best parts to create a mixed environment that is productive and nice to work with. I will present my approach to deployment of Django webapps controlled by supervisord, using Capistrano - an automated deployment tool belonging to the Ruby on Rails ecosystem - and my capistrano-offroad package that adapts Capistrano for non-Rails applications. I will also talk about release management process, based on the Git version control system, upon which the deployment setup is built.

Statistics

Views

Total Views
3,018
Views on SlideShare
2,869
Embed Views
149

Actions

Likes
5
Downloads
29
Comments
0

5 Embeds 149

http://www.setjam.com 95
http://www.slideshare.net 36
http://arvind.soup.io 9
http://leafnode.soup.io 5
http://japhy.soup.io 4

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Best Of Both Worlds Best Of Both Worlds Presentation Transcript

    • Best of both worlds Deploying Django projects with Capistrano and Supervisord Maciej Pasternacki SetJam, Inc. maciej@pasternacki.net Wednesday, May 26, 2010 1
    • Best of both worlds Should we only use Python tools with Django, or rather pick and mix best tools available, regardless of the ecosystem they come from? http://www.setjam.com/ Wednesday, May 26, 2010 2
    • Background SetJam.com: an online TV guide Started June 2009, public beta December 2009, incorporated February 2010 Agile & Open Source-based http://www.setjam.com/ Wednesday, May 26, 2010 3
    • What’s this talk about? Release management Automated deployment Upstream libraries Process management Scaling in cloud Centralized logging http://www.setjam.com/ Wednesday, May 26, 2010 4
    • Release management Weekly iterations Devel → Staging → Production All in git branches http://www.setjam.com/ Wednesday, May 26, 2010 5
    • Capistrano automating the deployment Separate, independent releases Mature & complete Full workflow Transactions and rollbacks http://www.setjam.com/ Wednesday, May 26, 2010 6
    • Capistrano directory layout releases YYYYMMDDHHMMSS 20100512210109 20100514101512 20100515085101 20100515101756 current log var/ solr_data shared processed_images local_settings.py … … http://www.setjam.com/ Wednesday, May 26, 2010 7
    • deploy:setup Prepares one or more servers for deployment. deploy:default Deploys your project. deploy:update Copies your project and updates the symlink. deploy:update_code Copies your project to the remote servers. strategy.deploy! deploy:finalize_update [internal] Touches up the released code. deploy:symlink Updates the symlink to the most recently deployed release deploy:restart Restarts your application. http:/ Transaction /www.setjam.com/ http://www.capify.org/index.php/Default_Deployment_Path Wednesday, May 26, 2010 8
    • Capistrano-offroad an open source pack of Capistrano utils Reset Capistrano’s railsy defaults Handful of utility functions Django and Supervisord recipes Want to improve! http://www.setjam.com/ Wednesday, May 26, 2010 9
    • Capistrano configuration samples: settings # Capistrano-offroad modules load 'lib/capistrano-offroad/offroad.rb' offroad_modules 'defaults', 'django', 'supervisord' # Server config server "staging.setjam.com", :web, :db, :primary => true role :app, "molly.setjam.com" group :staging_fetch, :backend set :deploy_to, "/srv/staging" set :deploy_user, "staging" set :deploy_group, "staging" set :python, "/opt/python26/bin/python2.6" set :django_project_subdirectory, "setjam" set :django_use_south, true http://www.setjam.com/ Wednesday, May 26, 2010 10
    • Capistrano configuration samples: dependencies depend :remote, :command, "make" depend :remote, :command, "s3cmd" […] depend :remote, :python_module, "PIL" depend :remote, :python_module, "flup" depend :remote, :python_module, "lxml" depend :remote, :python_module, "MySQLdb" http://www.setjam.com/ Wednesday, May 26, 2010 11
    • Capistrano configuration samples: tasks desc "[internal] finalize code update" task :finalize_update do run "ln -sfv #{shared_path}/local_settings*.py #{latest_release}/setjam/" run "make -C #{latest_release}/lib/ wait-for-cached download-cached" run "mv #{latest_release}/var #{latest_release}/var.orig" run "ln -sv #{shared_path}/var #{latest_release}/var" django_manage "synccompress", :roles => [:app, :web] _role_supervisord_conf :backend _role_supervisord_conf :app end desc "[internal] Wait until processes are reachable" task :wait, :roles => :app do run "#{current_path}/scripts/waitfcgi #{fcgi_port}" run "#{current_path}/scripts/waitfcgi #{fcgi_api_port}" end http://www.setjam.com/ Wednesday, May 26, 2010 12
    • Capistrano configuration samples: hooks after "deploy:setup", "setjam:setup" before "deploy:finalize_update", "setjam:finalize_update" before "deploy:symlink", "smoke:offline" after "deploy:start", "smoke" after "deploy:restart", "smoke" before "smoke", "setjam:wait" before "deploy:cleanup" do set :use_sudo, true end after "deploy:cleanup" do set :use_sudo, false end http://www.setjam.com/ Wednesday, May 26, 2010 13
    • Upstream libraries how to get them? System-wide? Project-wide (shared?) Release-wide! http://www.setjam.com/ Wednesday, May 26, 2010 14
    • Upstream libraries standard approach: virtualenv+pip pulling from upstream VC? add local patches? cache library set? non-python dependencies? http://www.setjam.com/ Wednesday, May 26, 2010 15
    • Upstream libraries current approach: Makefile Make tasks to get and patch libs Timestamp files for dependencies Cache versioned tarballs in S3 Symlinks to use downloaded stuff http://www.setjam.com/ Wednesday, May 26, 2010 16
    • Upstream libraries current approach: Makefile all: Django.stamp _submodules.stamp beautifulsoup.stamp boto.stamp concurrentloghandler.stamp oauth.stamp piston.stamp httplib2.stamp pyflakes.stamp python-memcached.stamp south.stamp test_coverage.stamp urllib3.stamp pyyaml.stamp feedparser.stamp unittest2.stamp chmod -R a+rX . south.stamp: south.release south-return_status.diff rm -rf south hg clone -r `head -1 south.release` http://bitbucket.org/andrewgodwin/south/ cd south && patch -p1 < ../south-return_status.diff date > $@ http://www.setjam.com/ Wednesday, May 26, 2010 17
    • Upstream libraries current approach: Makefile .PHONY: _submodules.stamp _submodules.stamp: cd .. && git submodule update --init date > $@ DJANGO_VERSION=1.1.1 DJANGO_BASENAME=Django-$(DJANGO_VERSION) $(DJANGO_BASENAME).tar.gz: wget -q -O $@ http://www.djangoproject.com/download/$(DJANGO_VERSION)/tarball/ touch $@ # needed - wget doesn't update timestamp properly Django.stamp: $(DJANGO_BASENAME).tar.gz Django-test_suite.3.diff Django-unittest2.diff rm -rf $(DJANGO_BASENAME) Django tar -xzf $< cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test_suite.3.diff cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test-client-cookie-fix.diff cd $(DJANGO_BASENAME) && patch -p1 < ../Django-unittest2.diff ln -s $(DJANGO_BASENAME) Django date > $@ http://www.setjam.com/ Wednesday, May 26, 2010 18
    • Supervisord process management Manages foreground processes Easy to use locally Easy to configure — .ini file Easy to extend Robust and reliable http://www.setjam.com/ Wednesday, May 26, 2010 19
    • Supervisord configuration sample [program:fcgi] command = python2.6 manage.py runfcgi daemonize=false … stopsignal = HUP directory = %(here)s/../setjam/ [program:api_fcgi] command = python2.6 manage.py runfcgi daemonize=false … stopsignal = HUP directory = %(here)s/../setjam/ environment = DJANGO_SETTINGS_MODULE=settings_api [program:classifier] command = python2.6 manage.py runreceiver classification directory = %(here)s/../setjam/ autostart = false process_name = classify-%(process_num)s numprocs = 3 http://www.setjam.com/ Wednesday, May 26, 2010 20
    • Scaling in cloud Few central servers Symmetrical processes Queuing to distribute work: django-sqs EC2 security groups to mark roles: capistrano-ec2group http://www.setjam.com/ Wednesday, May 26, 2010 21
    • Scaling in cloud deploying a new server Start new instance cap HOSTFILTER=… deploy:setup Copy local_settings.py cap HOSTFILTER=… deploy Maybe update central server config http://www.setjam.com/ Wednesday, May 26, 2010 22
    • Logging logging.handlers.SysLogHandler Central rsyslogd writes to MySQL phplogcon for log browsing Customized log reports from MySQL http://www.setjam.com/ Wednesday, May 26, 2010 23
    • http://www.setjam.com/ Wednesday, May 26, 2010 24
    • http://www.setjam.com/ Wednesday, May 26, 2010 25
    • Future work Monitoring and status - Nagios? Automate machine administration - Chef? Polish and gemify capistrano-offroad Rake instead of make for upstream libs Capistrano rendering configs from templates http://www.setjam.com/ Wednesday, May 26, 2010 26
    • Questions? More info: www.setjam.com/blog/ Code: github.com/mpasternacki/ More code: gist.github.com/mpasternacki/ E-mail: maciej@pasternacki.net Twitter: @mpasternacki http://www.setjam.com/ Wednesday, May 26, 2010 27