Pyramid Deployment and Maintenance    Carlos de la Guardia Plone Conference 2011
So, your application is ready and looking good
Time to deploy it and see how it holds up
Deployment with NginXMore information at:http://docs.pylonsproject.org/projects/pyramid_cookbook/dev/deployment/nginx.html...
Deployment with Apache and Mod_WSGIWe need to create an app that will call our application with configuration.Then we set ...
To paste or not to paste●   Paster is not considered a super fast wsgi server, but its good    enough for many use cases.●...
Putting all your eggs in the same basket
Your very own package index●   Create an egg directory inside an Apache web server    or similar and use a simple script t...
Using Buildout with Pyramid●   Buildout is a Python system for assembling    applications from multiple parts in a repeata...
Sample buildout[buildout]parts =   myapp   mkreleaseDevelop = src/mypackageindex = http://example.github.com/myapp/staging...
Supervisor●   Supervisor is a client/server system that allows    its users to monitor and control a number of    processe...
Sample supervisor configuration[inet_http_server]port=127.0.0.1:9001[supervisord]logfile=%(here)s/var/supervisord.loglogfi...
Useful maintenance tools
Scripting Pyramid●   Web applications usually expect to interact with a request    object of some kind and return a respon...
Writing the script●   To generate the proper environment we use    pyramid.paster.bootstrap, like this:    from pyramid.pa...
Exception logging with pyramid_exclog[loggers]keys = exc_logger[handlers]keys = exc_handler[formatters]keys = exc_formatte...
BackupsFor ZODB based applications:●   Collective.recipe.backupFor Postgres based applications:●   SQL dump●   File system...
Do not forget about staging
Thinking about staging●   A good deployment plan requires a good    staging strategy.●   Its important to be able to quick...
So much to do, so little timeOther deployment related tasks and decisions  ●   Caching  ●   Monitoring  ●   Failover  ●   ...
Case Study: KARL●   Open source web system for collaboration,    organizational intranets, and knowledge    management.●  ...
How is KARL deployed?●   Toolchain: Nginx, HAProxy, Paste, running    under Supervisor.●   Buildout for environment creati...
Thank You!       Email: cguardia@yahoo.com          IRC: #pyramid, #pylonshttp://groups.google.com/group/pylons-devel
Goodbye!
Upcoming SlideShare
Loading in...5
×

Pyramid Deployment and Maintenance

2,293

Published on

In this talk, Carlos de la Guardia shows how a Pyramid application can be deployed using a front end web server, like Apache or Nginx. He also covers how to automate deployment using buildout and a PyPI clone, and post-deployment creation of a variety of maintenance scripts and cron jobs that perform application specific tasks through Pyramid.

A link to audio of the presentation is here: http://2011ploneconference.sched.org/event/29a2f357905e4ab0fe3048c53bc1c94c

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,293
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Pyramid Deployment and Maintenance

  1. 1. Pyramid Deployment and Maintenance Carlos de la Guardia Plone Conference 2011
  2. 2. So, your application is ready and looking good
  3. 3. Time to deploy it and see how it holds up
  4. 4. Deployment with NginXMore information at:http://docs.pylonsproject.org/projects/pyramid_cookbook/dev/deployment/nginx.htmlupstream myapp-site { server 127.0.0.1:5000; server 127.0.0.1:5001;}server { server_name example.com; access_log /home/example/env/access.log; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 60s; proxy_send_timeout 90s; proxy_read_timeout 90s; proxy_buffering off; proxy_pass http://myapp-site; proxy_redirect off; }}
  5. 5. Deployment with Apache and Mod_WSGIWe need to create an app that will call our application with configuration.Then we set up apache to call this app.More information at:http://docs.pylonsproject.org/projects/pyramid/1.2/tutorials/modwsgi/index.html#pyramid.wsgifrom pyramid.paster import get_appapplication = get_app(/mydir/modwsgi/env/myapp/production.ini, main)#apache.confWSGIApplicationGroup %{GLOBAL}WSGIPassAuthorization OnWSGIDaemonProcess pyramid user=cguardia group=staff threads=4 python-path=/mydir/modwsgi/env/lib/python2.6/site-packagesWSGIScriptAlias /myapp /mydir/modwsgi/env/pyramid.wsgi<Directory /mydir/modwsgi/env> WSGIProcessGroup pyramid Order allow,deny Allow from all</Directory>
  6. 6. To paste or not to paste● Paster is not considered a super fast wsgi server, but its good enough for many use cases.● We already discussed mod_wsgi. If you are sure that you need something else, people on our mailing lists have reported success with uWSGI and gUnicorn.● If you are using ZODB with paster, consider using Jove to manage multiple sites and settings consistently: https://github.com/Pylons/jove● Other approaches include using gevent, twisted or eventlets.
  7. 7. Putting all your eggs in the same basket
  8. 8. Your very own package index● Create an egg directory inside an Apache web server or similar and use a simple script to generate the index in PyPI format.● Create an egg directory using a GitHub site and generate the index with the same script.● Install a PyPI clone like ClueReleaseManager, and configure your .pypirc file to point at the clone. Use setuptools to upload.● Jarn.mkrelease can be used to commit, tag and upload in one step: $ mkrelease -d mypypi src/my.package
  9. 9. Using Buildout with Pyramid● Buildout is a Python system for assembling applications from multiple parts in a repeatable manner.● Recipes are used to define what each part of the buildout will install and/or setup. There are many available recipes on PyPI.● A buildout can have different configurations. For example, deployment and production.● Buildout can be used to setup a Pyramid application and its dependencies easily.
  10. 10. Sample buildout[buildout]parts = myapp mkreleaseDevelop = src/mypackageindex = http://example.github.com/myapp/staging/index/[myapp]recipe = zc.recipe.eggeggs = some_dependency mypackageinterpreter = py[mkrelease]recipe = zc.recipe.eggeggs = jarn.mkreleasescripts = mkrelease
  11. 11. Supervisor● Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.● We can use supervisor to control our pyramid applications, either alone or together with other applications or services.● A very easy way to configure it is to simply add the supervisor egg to our buildout and include a configuration file.
  12. 12. Sample supervisor configuration[inet_http_server]port=127.0.0.1:9001[supervisord]logfile=%(here)s/var/supervisord.loglogfile_maxbytes=50MBlogfile_backups=10loglevel=infopidfile=%(here)s/var/supervisord.pid[rpcinterface:supervisor]supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface[supervisorctl]serverurl=http://127.0.0.1:9001[program:core]command = %(here)s/bin/paster serve %(here)s/src/inav2_core/development.iniredirect_stderr = true
  13. 13. Useful maintenance tools
  14. 14. Scripting Pyramid● Web applications usually expect to interact with a request object of some kind and return a response.● Importing web application code from a command line script will not work when it depends on a request.● Pyramid offers a facility for generating an environment similar to what the application would get from a real request.● This makes it possible to easily write scripts for tasks like updating databases, create sample content or initialize application services.
  15. 15. Writing the script● To generate the proper environment we use pyramid.paster.bootstrap, like this: from pyramid.paster import bootstrap env = bootstrap(/path/to/my/development.ini) print env[request].route_url(home)● We get back a dictionary with: – sequest – app – root – registry – closer
  16. 16. Exception logging with pyramid_exclog[loggers]keys = exc_logger[handlers]keys = exc_handler[formatters]keys = exc_formatter[logger_exc_logger]level = ERRORhandlers = exc_handlerqualname = exc_logger[handler_exc_handler]class = handlers.SMTPHandlerargs = ((localhost, 25), from@example.com, [to@example.com], myapp Exception)level = ERRORformatter = exc_formatter[formatter_exc_formatter]format = %(asctime)s %(message)s
  17. 17. BackupsFor ZODB based applications:● Collective.recipe.backupFor Postgres based applications:● SQL dump● File system level backup● Continuous archiving
  18. 18. Do not forget about staging
  19. 19. Thinking about staging● A good deployment plan requires a good staging strategy.● Its important to be able to quickly deploy changes to staging before trying them in production.● Ideally, multiple branches could be deployed separately.● Take a look at octomotron: https://github.com/chrisrossi/octomotron
  20. 20. So much to do, so little timeOther deployment related tasks and decisions ● Caching ● Monitoring ● Failover ● Replication ● Automated deployment ● More...
  21. 21. Case Study: KARL● Open source web system for collaboration, organizational intranets, and knowledge management.● Mission-critical application used by organizations such as Open Society Foundations and Oxfam.● 5000+ users (no anonymous access).● More than 75,000 Pages of content.● Nearly 100,000 lines of code.
  22. 22. How is KARL deployed?● Toolchain: Nginx, HAProxy, Paste, running under Supervisor.● Buildout for environment creation.● Custom package index stored on GitHub.● 1 App Server, 1 DB Server.● Uses ZODB with RelStorage and PGTextIndex.● Deployment is done with a Python script that creates a parallel environment, backs up and evolves data, then restarts the instance.
  23. 23. Thank You! Email: cguardia@yahoo.com IRC: #pyramid, #pylonshttp://groups.google.com/group/pylons-devel
  24. 24. Goodbye!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×