Your SlideShare is downloading. ×
Python & Devops: Your own heroku
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Python & Devops: Your own heroku

2,354
views

Published on

Using python and uWSGI for agile deployment, effortless :) …

Using python and uWSGI for agile deployment, effortless :)

Published in: Technology

0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,354
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
38
Comments
0
Likes
12
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Python & DevOps Your own heroku jib.li
  • 2. Chakib Benziane @sp4ke● Freelance Full Stack Developer / DevOps● CTO & CoFounder at Jib.li● Python Enthusiast
  • 3. Summary1. Environment & Stack2. Agile deployment with uWSGI
  • 4. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages
  • 5. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages = Django + MongoDB + Github
  • 6. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages = Django + MongoDB + Github gevent-socketio, zmq, celery, AWS boto ...
  • 7. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP
  • 8. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP ● But they have RVM !
  • 9. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP ● But they have RVM ! ● Pythonbrew: utahta/pythonbrew.git
  • 10. 1. Environment & Stack● Pythonbrew: utahta/pythonbrew.git ○ Compile system independent pythons $ pythonbrew install 2.7.3 $ pythonbrew use 2.7.3 $ pythonbrew list && which python # pythonbrew pythons Python-2.7.3 (*) /home/spike/.pythonbrew/pythons/Python-2.7.3/bin/python
  • 11. 1. Environment & Stack● Pythonbrew: utahta/pythonbrew.git ○ Easy management of virtualenvs $ pythonbrew venv create jibli $ pythonbrew venv use jibli && which python && which pip # Using `jibli` environment # To leave an environment, simply run `deactivate` /home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/python /home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/pip
  • 12. 1. Environment & Stack Environment bootstrap: git clone jibli/project && cd project pythonbrew create venv jibli && pythonbrew activate jibli pip install -r dependencies.txt
  • 13. 1. Environment & Stack Environment bootstrap: git clone jibli/project && cd project pythonbrew create venv jibli && pythonbrew activate jibli pip install -r dependencies.txt --download-cache=CACHE
  • 14. 1. Environment & Stack pip & dependencies.txt: package-foo dont package-bar==4.2 do
  • 15. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo
  • 16. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package
  • 17. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package git+https://github.com/user/repo@branch
  • 18. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package git+https://github.com/user/repo@branch ● Quick dependencies update pip freeze > dependencies.txt
  • 19. 1. Environment & Stack MongoDB ● NoSQL, Schemaless, Document Oriented ● BSON data format ● Advantage: Python Dict -> JSON ● Good Python API pymongo
  • 20. 1. Environment & Stack MongoDB● MongoDB Javascript Console $ mongo jibli MongoDB shell version: 2.0.6 connecting to: jibli > db.users.find( {profil.age: 10} );
  • 21. 1. Environment & Stack MongoDB● MongoDB Javascript Console $ mongo jibli MongoDB shell version: 2.0.6 connecting to: jibli > db.users.find( {profil.age: 10} );● PyMongo equivalent u = pymongo.Connection(host=localhost, port=27017)[jibli][users] u.find( {profil.age: 10} )
  • 22. 2. Agile deployment with uWSGI Local development● Git branch feature● Unit test● Implement● Test on local server (./manage.py runserver)● Commit and merge on master branch
  • 23. 2. Agile deployment with uWSGI Agile deployment ● Many features require a production like environment: ○ OAuth Authentication and Social Networks ○ Async tasks, Celery (notifications, crons ... ) ○ Push Notifications ○ Hard to clone a production environment in local
  • 24. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature
  • 25. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push
  • 26. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com
  • 27. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com ○ Remote access to the deployed app from local shell with a production environment (ie. restart, upgrade, ipython, mongo shell ...)
  • 28. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com ○ Remote access to the deployed app from local shell with a production environment (ie. restart, upgrade, ipython, mongo shell ...) ○ Once satisfied merge on master and push on prod
  • 29. 2. Agile deployment with uWSGI Solution● Nginx● Github● Fabric+● uWSGI
  • 30. 2. Agile deployment with uWSGI Solution● Nginx● Github● Fabric+● uWSGI = your heroku like solution
  • 31. 2. Agile deployment with uWSGI uWSGI● Create developement stacks● Host application clusters
  • 32. 2. Agile deployment with uWSGI uWSGI ● how ? Web Server Django (nginx) uWSGI App WSGI Interface also: FastCGI,CGI, PHP, Rack, ...
  • 33. 2. Agile deployment with uWSGI uWSGI ● how ? Django App Web Server Django (nginx) uWSGI App Load balancing Django App
  • 34. 2. Agile deployment with uWSGI uWSGI ● how ? Django App1 Web Server Django (nginx) uWSGI App2 Router app1.dev.com app2.dev.com app3.dev.com Proxy Flask App3
  • 35. 2. Agile deployment with uWSGI Django uWSGI App1 ● how ? Django App2 Web Server Flask (nginx) uWSGI App3 Emperor app1.dev.com app2.dev.com Massive app app3.dev.com *.dev.com deployment App n
  • 36. 2. Agile deployment with uWSGI uWSGI Emperor ● Event based dynamic handling of applications (Vassals)Default: ● Scan for config files in directories (.ini, .xml, .yml, .json ... ) ● dir:// & glob:// for conf files monitoring ● Much more plugins available (mongodb, ampq, ldap ... )
  • 37. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor /opt/apps/*/*.ini
  • 38. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor "/opt/apps/*/*.ini" One does not simply use glob patterns !
  • 39. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor "/opt/apps/*/*.ini" Example: ○ New file "/opt/apps/appn/uwsgi.ini" ● Spawn vassal ○ File modified ● Restart vassal ○ File removed ● Kill vassal ○ Emperor dies ● All vassals die with him
  • 40. 2. Agile deployment with uWSGI ● Create a conf file for each deployed app ? /opt/apps/app1/uwsgi.ini /opt/apps/app2/uwsgi.ini /opt/apps/appn/uwsgi.ini
  • 41. 2. Agile deployment with uWSGI ● Create a conf file for each deployed app ? /opt/apps/app1/uwsgi.ini /opt/apps/app2/uwsgi.ini /opt/apps/appn/uwsgi.ini ● ln -s ○ Use template conf files /opt/apps/template ln -s /opt/apps/template /opt/apps/app1/app1.ini
  • 42. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings chdir = %(djangoproject) module = uwsgi_app socket = 127.0.0.1:0 master = true processes = 1 idle = 300 subscribe-to = 127.0.0.1:9999:%n.dev.com logto = %d/log/uwsgi.log
  • 43. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ ● Use variables like here djangoproject ● Magic variables : ○ %d - Absolute path to configuration file ○ %n - Name of configuration file without extension
  • 44. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv
  • 45. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv ● Python search paths (You can repeat this one)
  • 46. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv ● Python search paths (You can repeat this one) ● Custom environment variables
  • 47. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] chdir = %(djangoproject) module = uwsgi_app ● Which module to run when starting application ○ django.core.handlers.wsgi:WSGIHandler() ● Best spot to run your custom scripts and setup environment before launching application ○ ie. compile static, zMQ sockets, syncdb ...
  • 48. 2. Agile deployment with uWSGI Up until now we can: ○ Git push origin feature ○ Clone feature in remote /opt/apps/feature ○ Prepare dirs structure ○ Create venv & install dependencies ○ Symllink to the uWSGI template file ○ uWSGI emperor launches feature app We still need to: ○ access our feature using subdomains ● feature.dev.com
  • 49. 2. Agile deployment with uWSGI FastRouter ○ Proxy/Load Balancing/Router ○ Speaks uWSGI protocol ○ Unlimited setup possibilities ○ Key/Value store
  • 50. 2. Agile deployment with uWSGI FastRouter ○ Proxy/Load Balancing/Router ○ Speaks uWSGI protocol ○ Unlimited setup possibilities ○ Key/Value store Example: uwsgi --fastrouter /tmp/fastrouter.socket -- fastrouter-subscription-server 127.0.0.1:9999
  • 51. 2. Agile deployment with uWSGI Always use unix sockets instead of localhost tcp
  • 52. 2. Agile deployment with uWSGI Nginx server { listen 80; server_name dev.com *.dev.com; location / { include /etc/nginx/uwsgi_params; uwsgi_param UWSGI_FASTROUTER_KEY $host; uwsgi_pass unix: /tmp/fastrouter.socket; } }
  • 53. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] ... socket = 127.0.0.1:0 subscribe-to = 127.0.0.1:9999:%n.dev.com
  • 54. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] ... socket = 127.0.0.1:0 subscribe-to = 127.0.0.1:9999:%n.dev.com /opt/apps/feature1/feature1.ini
  • 55. 2. Agile deployment with uWSGI Request on feature1.dev.com nginx Fastrouteur + Subscription server { feature1.dev.com : 127.0.0.1:n1, feautre2.dev.com : 127.0.0.1:n2, ... } uWSGI emperor feature1 feature2
  • 56. 2. Agile deployment with uWSGI HTOP Deploying feature1 ├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini │ │ ├─ /usr/local/bin/uwsgi #fastrouter │ │ └─ /usr/local/bin/uwsgi #master │ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini
  • 57. 2. Agile deployment with uWSGI HTOP Deploying feature2 ├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini │ │ ├─ /usr/local/bin/uwsgi #fastrouter │ │ └─ /usr/local/bin/uwsgi #master │ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini │ │ └─uwsgi --ini /opt/apps/feature2/frature2.ini
  • 58. End twitter: @sp4ke email: spike@jib.lislides: sp4ke.com/pythondevops