Python & Devops: Your own heroku

3,321 views

Published on

Using python and uWSGI for agile deployment, effortless :)

Published in: Technology

Python & Devops: Your own heroku

  1. 1. Python & DevOps Your own heroku jib.li
  2. 2. Chakib Benziane @sp4ke● Freelance Full Stack Developer / DevOps● CTO & CoFounder at Jib.li● Python Enthusiast
  3. 3. Summary1. Environment & Stack2. Agile deployment with uWSGI
  4. 4. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages
  5. 5. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages = Django + MongoDB + Github
  6. 6. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages = Django + MongoDB + Github gevent-socketio, zmq, celery, AWS boto ...
  7. 7. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP
  8. 8. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP ● But they have RVM !
  9. 9. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP ● But they have RVM ! ● Pythonbrew: utahta/pythonbrew.git
  10. 10. 1. Environment & Stack● Pythonbrew: utahta/pythonbrew.git ○ Compile system independent pythons $ pythonbrew install 2.7.3 $ pythonbrew use 2.7.3 $ pythonbrew list && which python # pythonbrew pythons Python-2.7.3 (*) /home/spike/.pythonbrew/pythons/Python-2.7.3/bin/python
  11. 11. 1. Environment & Stack● Pythonbrew: utahta/pythonbrew.git ○ Easy management of virtualenvs $ pythonbrew venv create jibli $ pythonbrew venv use jibli && which python && which pip # Using `jibli` environment # To leave an environment, simply run `deactivate` /home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/python /home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/pip
  12. 12. 1. Environment & Stack Environment bootstrap: git clone jibli/project && cd project pythonbrew create venv jibli && pythonbrew activate jibli pip install -r dependencies.txt
  13. 13. 1. Environment & Stack Environment bootstrap: git clone jibli/project && cd project pythonbrew create venv jibli && pythonbrew activate jibli pip install -r dependencies.txt --download-cache=CACHE
  14. 14. 1. Environment & Stack pip & dependencies.txt: package-foo dont package-bar==4.2 do
  15. 15. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo
  16. 16. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package
  17. 17. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package git+https://github.com/user/repo@branch
  18. 18. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package git+https://github.com/user/repo@branch ● Quick dependencies update pip freeze > dependencies.txt
  19. 19. 1. Environment & Stack MongoDB ● NoSQL, Schemaless, Document Oriented ● BSON data format ● Advantage: Python Dict -> JSON ● Good Python API pymongo
  20. 20. 1. Environment & Stack MongoDB● MongoDB Javascript Console $ mongo jibli MongoDB shell version: 2.0.6 connecting to: jibli > db.users.find( {profil.age: 10} );
  21. 21. 1. Environment & Stack MongoDB● MongoDB Javascript Console $ mongo jibli MongoDB shell version: 2.0.6 connecting to: jibli > db.users.find( {profil.age: 10} );● PyMongo equivalent u = pymongo.Connection(host=localhost, port=27017)[jibli][users] u.find( {profil.age: 10} )
  22. 22. 2. Agile deployment with uWSGI Local development● Git branch feature● Unit test● Implement● Test on local server (./manage.py runserver)● Commit and merge on master branch
  23. 23. 2. Agile deployment with uWSGI Agile deployment ● Many features require a production like environment: ○ OAuth Authentication and Social Networks ○ Async tasks, Celery (notifications, crons ... ) ○ Push Notifications ○ Hard to clone a production environment in local
  24. 24. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature
  25. 25. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push
  26. 26. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com
  27. 27. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com ○ Remote access to the deployed app from local shell with a production environment (ie. restart, upgrade, ipython, mongo shell ...)
  28. 28. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com ○ Remote access to the deployed app from local shell with a production environment (ie. restart, upgrade, ipython, mongo shell ...) ○ Once satisfied merge on master and push on prod
  29. 29. 2. Agile deployment with uWSGI Solution● Nginx● Github● Fabric+● uWSGI
  30. 30. 2. Agile deployment with uWSGI Solution● Nginx● Github● Fabric+● uWSGI = your heroku like solution
  31. 31. 2. Agile deployment with uWSGI uWSGI● Create developement stacks● Host application clusters
  32. 32. 2. Agile deployment with uWSGI uWSGI ● how ? Web Server Django (nginx) uWSGI App WSGI Interface also: FastCGI,CGI, PHP, Rack, ...
  33. 33. 2. Agile deployment with uWSGI uWSGI ● how ? Django App Web Server Django (nginx) uWSGI App Load balancing Django App
  34. 34. 2. Agile deployment with uWSGI uWSGI ● how ? Django App1 Web Server Django (nginx) uWSGI App2 Router app1.dev.com app2.dev.com app3.dev.com Proxy Flask App3
  35. 35. 2. Agile deployment with uWSGI Django uWSGI App1 ● how ? Django App2 Web Server Flask (nginx) uWSGI App3 Emperor app1.dev.com app2.dev.com Massive app app3.dev.com *.dev.com deployment App n
  36. 36. 2. Agile deployment with uWSGI uWSGI Emperor ● Event based dynamic handling of applications (Vassals)Default: ● Scan for config files in directories (.ini, .xml, .yml, .json ... ) ● dir:// & glob:// for conf files monitoring ● Much more plugins available (mongodb, ampq, ldap ... )
  37. 37. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor /opt/apps/*/*.ini
  38. 38. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor "/opt/apps/*/*.ini" One does not simply use glob patterns !
  39. 39. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor "/opt/apps/*/*.ini" Example: ○ New file "/opt/apps/appn/uwsgi.ini" ● Spawn vassal ○ File modified ● Restart vassal ○ File removed ● Kill vassal ○ Emperor dies ● All vassals die with him
  40. 40. 2. Agile deployment with uWSGI ● Create a conf file for each deployed app ? /opt/apps/app1/uwsgi.ini /opt/apps/app2/uwsgi.ini /opt/apps/appn/uwsgi.ini
  41. 41. 2. Agile deployment with uWSGI ● Create a conf file for each deployed app ? /opt/apps/app1/uwsgi.ini /opt/apps/app2/uwsgi.ini /opt/apps/appn/uwsgi.ini ● ln -s ○ Use template conf files /opt/apps/template ln -s /opt/apps/template /opt/apps/app1/app1.ini
  42. 42. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings chdir = %(djangoproject) module = uwsgi_app socket = 127.0.0.1:0 master = true processes = 1 idle = 300 subscribe-to = 127.0.0.1:9999:%n.dev.com logto = %d/log/uwsgi.log
  43. 43. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ ● Use variables like here djangoproject ● Magic variables : ○ %d - Absolute path to configuration file ○ %n - Name of configuration file without extension
  44. 44. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv
  45. 45. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv ● Python search paths (You can repeat this one)
  46. 46. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv ● Python search paths (You can repeat this one) ● Custom environment variables
  47. 47. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] chdir = %(djangoproject) module = uwsgi_app ● Which module to run when starting application ○ django.core.handlers.wsgi:WSGIHandler() ● Best spot to run your custom scripts and setup environment before launching application ○ ie. compile static, zMQ sockets, syncdb ...
  48. 48. 2. Agile deployment with uWSGI Up until now we can: ○ Git push origin feature ○ Clone feature in remote /opt/apps/feature ○ Prepare dirs structure ○ Create venv & install dependencies ○ Symllink to the uWSGI template file ○ uWSGI emperor launches feature app We still need to: ○ access our feature using subdomains ● feature.dev.com
  49. 49. 2. Agile deployment with uWSGI FastRouter ○ Proxy/Load Balancing/Router ○ Speaks uWSGI protocol ○ Unlimited setup possibilities ○ Key/Value store
  50. 50. 2. Agile deployment with uWSGI FastRouter ○ Proxy/Load Balancing/Router ○ Speaks uWSGI protocol ○ Unlimited setup possibilities ○ Key/Value store Example: uwsgi --fastrouter /tmp/fastrouter.socket -- fastrouter-subscription-server 127.0.0.1:9999
  51. 51. 2. Agile deployment with uWSGI Always use unix sockets instead of localhost tcp
  52. 52. 2. Agile deployment with uWSGI Nginx server { listen 80; server_name dev.com *.dev.com; location / { include /etc/nginx/uwsgi_params; uwsgi_param UWSGI_FASTROUTER_KEY $host; uwsgi_pass unix: /tmp/fastrouter.socket; } }
  53. 53. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] ... socket = 127.0.0.1:0 subscribe-to = 127.0.0.1:9999:%n.dev.com
  54. 54. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] ... socket = 127.0.0.1:0 subscribe-to = 127.0.0.1:9999:%n.dev.com /opt/apps/feature1/feature1.ini
  55. 55. 2. Agile deployment with uWSGI Request on feature1.dev.com nginx Fastrouteur + Subscription server { feature1.dev.com : 127.0.0.1:n1, feautre2.dev.com : 127.0.0.1:n2, ... } uWSGI emperor feature1 feature2
  56. 56. 2. Agile deployment with uWSGI HTOP Deploying feature1 ├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini │ │ ├─ /usr/local/bin/uwsgi #fastrouter │ │ └─ /usr/local/bin/uwsgi #master │ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini
  57. 57. 2. Agile deployment with uWSGI HTOP Deploying feature2 ├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini │ │ ├─ /usr/local/bin/uwsgi #fastrouter │ │ └─ /usr/local/bin/uwsgi #master │ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini │ │ └─uwsgi --ini /opt/apps/feature2/frature2.ini
  58. 58. End twitter: @sp4ke email: spike@jib.lislides: sp4ke.com/pythondevops

×