Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Development environment 
for Django project - my way 
! 
Robert Lujo, 2014
about me 
• software 
• professionally 17 y. 
• python since 2.0, django since 0.96 
• freelancer 
• more info -> linkedin
“Software Development 
Environment” is …? 
I found no unique / unified answer …
Software Development 
Environment (1) 
refers to a server tier designated to a specific stage in a 
release process. 
List...
Software Development 
Environment (2) 
a collection of procedures and tools for developing, testing 
and debugging an appl...
Integrated development 
environment - IDE 
! 
software application that provides comprehensive facilities to 
computer pro...
Software Development 
Environment (3) 
contains everything required by a team to build and 
deploy software-intensive syst...
What is “everything”? 
Software development lifecycle (SDLC) 
• Planning 
• Designing 
• Implementation, testing and docum...
What about “a team”? 
Interacts with: 
• Software project management - organisation, 
coordination, covers: risk, change, ...
My priorities 
1. implement at least 90% of the most important 
requested features 
2. shortest possible time needed for “...
! 
now, the real stuff 
! 
btw. where has Django disappeared?
Initial setup - proj layout 
~/env 
├── proj-1 
├── cli-1 
├── … 
├── biz 
├── … 
└── proj-N
Initial setup - proj layout 
~/env/biz 
├── archive 
├── biz-web (GIT) 
│ ├── ... (to be continued) 
├── biz-data(GIT) 
├─...
Initial setup - python proj layout 
~/env/biz/biz-web (GIT) 
├── bin 
├── doc (sphinx,rst: readme,todo, 
│ install, ...) 
...
Initial setup - django proj layout 
~/env/biz/biz-web/src 
├── bizproj (global project folder - formats, settings) 
├── bi...
Initial setup - django settings 
~/env/biz/biz-web/src/bizproj/settings 
! 
base.py 
from .secret import * 
! 
dev.py 
dev...
Initial setup - .gitignore 
! 
~/env/biz/biz-web/.gitignore 
… 
pybiz 
var 
src/media/* 
secret.py 
fabfile_settings.py 
*...
File and directory naming 
• very careful in naming ! 
• naming convention - inherit existing + 
adaptations! 
• very care...
Project dependencies 
• as small as possible - more components -> more complex -> more integration/ 
upgrade/other issues ...
Project dependencies 
• solutions 
1. make a script to setup everything from the 
scratch 
2. install manually/reuse exist...
Project dependencies 
• make a shell script 
• two types of dependencies 
• pip installable 
pip install -r reqs/dev.txt 
...
Project deps - pip requirements 
! 
~/env/biz/biz-web/reqs 
! 
base.txt : 
django>=1.5,<1.6 
pillow 
reportlab 
johnny-cac...
Project deps - shell script 
! 
non-PIP installables 
! 
~/env/biz/biz-web/bin/biz # chmod u+x 
! 
case "$1" in 
env) 
if ...
Project deps - shell script 
! 
PIP installables 
! 
(continued …) 
! 
if [ -f $BIZ_HOME/pybiz/bin/activate ]; then 
sourc...
Project deps - shell script 
! 
check services 
! 
echo "== Is redis running?" 
ps ax | grep -v grep | grep redis 
!! 
ech...
Project management shell script 
• why? 
• to have one central place where to do project management 
• what? everything th...
Project management shell script 
! 
# biz 
! 
Usage: 
! 
*env - will initialize dev environment 
! 
redis - start redis in...
Editor: ViM 
• although steep learning curve, is: 
• very adaptable 
• unique features philosophy 
• not in the way, very ...
• personal preference: 
Editor: ViM 
• I don’t like plugins too much, only .vimrc 
• when working on the server to have th...
Development on the server 
• with project shell script one can do directly on 
the server 
• devenv setup/upgrade 
• webap...
File browser 
• personal preference: 
• I like side by side file browsers 
• usual file operations: create folders, check ...
Idea evaluation 
• evaluate a new idea or to test concept 
• ipython / django shell_plus 
• use reload: 
from biz import m...
• I use a lot 
Debugging 
• plain-old pdb or sometimes ipdb 
• conditional breakpoints: 
if i==500: 
import pdb; pdb.set_t...
• Unit tests 
• integration tests 
Testing 
• “click-testing” - local development server 
• automating http requests to st...
Deployment 
• must be simple, automated, fast and reliable 
• usually: 
• setup environment 
• git push+pull / collectstat...
Philosophy 
• “Unix philosophy by Mike Gancarz” filtered: 
Make each program do one thing well. 
Use software leverage to ...
Thanks for the patience! 
Questions? 
! 
robert.lujo@gmail.com 
@trebor74hr
Upcoming SlideShare
Loading in …5
×

Django dev-env-my-way

773 views

Published on

This talk will try to cover the most important techniques and best practices used when creating Django web application.

Overview of the topics covered:
- development general principles and goals
- python/django project initial setup - project layout, git&venv&pip&shell, settings
- central project shell command - contains all commands to manage project
- "IDE" - editor & shell
- edit/run/test cycle
- deploy/test-remotely cycle

Disclaimer: techniques and practices presented are current AUTHOR'S optimal choice used for usual django project.

Published in: Software
  • Login to see the comments

  • Be the first to like this

Django dev-env-my-way

  1. 1. Development environment for Django project - my way ! Robert Lujo, 2014
  2. 2. about me • software • professionally 17 y. • python since 2.0, django since 0.96 • freelancer • more info -> linkedin
  3. 3. “Software Development Environment” is …? I found no unique / unified answer …
  4. 4. Software Development Environment (1) refers to a server tier designated to a specific stage in a release process. List of tiers: Local, Development, Integration, Test/QA, UAT, Stage/Pre-production, Production/Live Hm…! https://en.wikipedia.org/wiki/Development_environment_ %28software_development_process%29
  5. 5. Software Development Environment (2) a collection of procedures and tools for developing, testing and debugging an application or program. Getting closer …! http://www.techopedia.com/definition/16376/development-environment
  6. 6. Integrated development environment - IDE ! software application that provides comprehensive facilities to computer programmers for software development. … source code editor, build automation tools and a debugger... Intelligent code completion features ... compiler, interpreter, ... version control system and various tools are integrated to simplify the construction of a GUI ... class browser, an object browser, and a class hierarchy diagram. Closer and closer …! https://en.wikipedia.org/wiki/Integrated_development_environment
  7. 7. Software Development Environment (3) contains everything required by a team to build and deploy software-intensive systems (where software is an essential and indispensable element) I approve!! http://www.ibm.com/developerworks/rational/library/define-scope-development- environment/index.html
  8. 8. What is “everything”? Software development lifecycle (SDLC) • Planning • Designing • Implementation, testing and documenting • Deployment and maintenance https://en.wikipedia.org/wiki/Software_development !
  9. 9. What about “a team”? Interacts with: • Software project management - organisation, coordination, covers: risk, change, software configuration and release management https://en.wikipedia.org/wiki/Software_project_management • Software development process a.k.a. software development methodology - e.g. waterfall, agile … https://en.wikipedia.org/wiki/Software_development_process
  10. 10. My priorities 1. implement at least 90% of the most important requested features 2. shortest possible time needed for “from request to deploy” cycle 3. low-cost scaling - human/machines - initial setup of DevEnv or running system 4. ensure high quality deliverables
  11. 11. ! now, the real stuff ! btw. where has Django disappeared?
  12. 12. Initial setup - proj layout ~/env ├── proj-1 ├── cli-1 ├── … ├── biz ├── … └── proj-N
  13. 13. Initial setup - proj layout ~/env/biz ├── archive ├── biz-web (GIT) │ ├── ... (to be continued) ├── biz-data(GIT) ├── biz-ie (GIT) ├── doc -> ln -s ~/Dropbox/…/biz/doc └── temp
  14. 14. Initial setup - python proj layout ~/env/biz/biz-web (GIT) ├── bin ├── doc (sphinx,rst: readme,todo, │ install, ...) ├── pybiz (venv python installation) ├── reqs (base.txt, dev.txt) ├── src │ ├── (to be continued) │ ... └── var ├── biz.sqlite ├── biz-app.log ├── biz-jobs.log └── bulk-data
  15. 15. Initial setup - django proj layout ~/env/biz/biz-web/src ├── bizproj (global project folder - formats, settings) ├── bizjobs │ ├── ... ├── bizapp │ ├── management / commands │ ├── migrations │ ├── scripts │ ├── templates │ │ ├── biz │ │ │ ├── components │ │ │ ├── backoffice │ │ │ ├── protected │ │ │ ├── public │ │ │ └── trial │ │ ├── admin │ │ ├── rosetta │ └── templatetags ├── locale (de/en) ├── media └── static (js, css, images, libs, fonts) # to bizapp?
  16. 16. Initial setup - django settings ~/env/biz/biz-web/src/bizproj/settings ! base.py from .secret import * ! dev.py dev_demo.py ! test.py ! secret.py # not in git repo EMAIL_HOST_USER = "..." DATABASES = { ! secrete.py.template # is in git repo ! dbrouter.py testrunner.py
  17. 17. Initial setup - .gitignore ! ~/env/biz/biz-web/.gitignore … pybiz var src/media/* secret.py fabfile_settings.py *.pyc *.swp *.pickle *.sqlite *.bak …
  18. 18. File and directory naming • very careful in naming ! • naming convention - inherit existing + adaptations! • very careful in choosing folder/structure! • bigger files better than many files! • less files/folders is better than more!
  19. 19. Project dependencies • as small as possible - more components -> more complex -> more integration/ upgrade/other issues • choose very carefully • proven, stable, I/team is familiar with • easily installable/upgradeable • adaptable (when possible) • when choosing new • read documentation/read reviews/choose several/experiment/make decision • bigger community -> better
  20. 20. Project dependencies • solutions 1. make a script to setup everything from the scratch 2. install manually/reuse existing images and then freeze state in VM, Vagrant, Docker image 3. combine both
  21. 21. Project dependencies • make a shell script • two types of dependencies • pip installable pip install -r reqs/dev.txt • pip non-installable • set of shell commands to install everything that is missing • Problem: different OS-es, e.g. OSX, Linux CentOS? if [ $(uname) == 'Linux' ] …
  22. 22. Project deps - pip requirements ! ~/env/biz/biz-web/reqs ! base.txt : django>=1.5,<1.6 pillow reportlab johnny-cache markdown elasticsearch beautifulsoup4 lxml requests celery[redis] … ! dev.txt : -r base.txt south django_extensions django-debug-toolbar django-rosetta # for translation mock ipdb …
  23. 23. Project deps - shell script ! non-PIP installables ! ~/env/biz/biz-web/bin/biz # chmod u+x ! case "$1" in env) if [ ! -d $BIZ_HOME/pybiz ]; then cd $BIZ_HOME sudo apt-get install python-devel sudo apt-get install python-pip sudo apt-get install python-xml sudo apt-get install elasticsearch ... pip install virtualenv virtualenv pybiz fi ! (to be continued …)
  24. 24. Project deps - shell script ! PIP installables ! (continued …) ! if [ -f $BIZ_HOME/pybiz/bin/activate ]; then source $BIZ_HOME/pybiz/bin/activate echo "pip=`which pip`, python=`which python`” ! pip install -q --requirement=$BIZ_HOME/reqs/dev.txt ! cd $BIZ_HOME/src/bizapp export PYTHONPATH=$PYTHONPATH:$BIZ_HOME/src echo "== env 'BIZ' initialized, checking base packages" python -c"import django; print(' Django - ok v%s' % django.get_version())" python -c"import redis ;print(' redis - ok')" python -c"import celery ;print(' celery - ok')" python -c"import bizappp ;print(' biz app - ok’)" … fi ! (to be continued …)
  25. 25. Project deps - shell script ! check services ! echo "== Is redis running?" ps ax | grep -v grep | grep redis !! echo "== Is elasticsearch running?" ps ax | grep -v grep | grep elasticsearch | awk "{print $1, $2, $3, $4, $5, "..."}" ! …
  26. 26. Project management shell script • why? • to have one central place where to do project management • what? everything that needs automation • setup initial environment, update environment • open editor/IDE • run tests • deployment, remote management
  27. 27. Project management shell script ! # biz ! Usage: ! *env - will initialize dev environment ! redis - start redis in background elas - start elasticsearch in background cel - start celery worker in background ! doc-update/du - will update doc/build/html wiki doc doc - will open doc/build/html/index.html - home of wiki ! log2 - will tail app log (if located in local var directory) ! run/n - django-admin runserver django/d - django-admin sp - django-admin shell_plus ! test-ping/tp - ping production test/t - django-admin test --settings=test_settings trans-make|tm - TRANSLATION - calls makemessages ! r/remote - will call fabric script for deployment and remote operations ! migrate|mig - do south migration - only dev system mig-auto - create new south auto-migration - only dev system ! browse|b - browse to localhost:8000 issues - browse to opened issues on bitbucket ! * - should be called with source/.
  28. 28. Editor: ViM • although steep learning curve, is: • very adaptable • unique features philosophy • not in the way, very effective • personal preference: • I don’t feel comfortable in IDE • I like to be close to “bare metal” • to use editor designed for source code editing
  29. 29. • personal preference: Editor: ViM • I don’t like plugins too much, only .vimrc • when working on the server to have the same possibilities • configuration • server scp ~/.vimrc user@server:/home/user/ • local development machine ln -s ~/Dropbox/env/.vimrc ~ • flavors • gVim / MacVim - GUI, ViM - on every server
  30. 30. Development on the server • with project shell script one can do directly on the server • devenv setup/upgrade • webapp setup/upgrade • vim & shell-script -> do “live” development • run dev server
  31. 31. File browser • personal preference: • I like side by side file browsers • usual file operations: create folders, check files, search for files, zip etc. • easy integration with editor (F4) • not found “one-to-rule-them-all” • TotalCommander, FreeCommander, DoubleCmd …
  32. 32. Idea evaluation • evaluate a new idea or to test concept • ipython / django shell_plus • use reload: from biz import models models.test_something() # change models and save reload(models) models.test_something() • next step is - create a module/function
  33. 33. • I use a lot Debugging • plain-old pdb or sometimes ipdb • conditional breakpoints: if i==500: import pdb; pdb.set_trace() • conditional breakpoints (2): try: bad_bad_code() except: import pdb; pdb.set_trace() • server side - logging, send emails on error logger.error(“this will be send by email) logger.error(“in try/except - will be send by email with traceback”)
  34. 34. • Unit tests • integration tests Testing • “click-testing” - local development server • automating http requests to staging/production servers (from curl, nagios to dedicated services) • “click-testing” - staging/production servers
  35. 35. Deployment • must be simple, automated, fast and reliable • usually: • setup environment • git push+pull / collectstatic / restart server • analyse logs • run integration tests on server instance • example: git ci -am”very important fix” biz r test upgrade prod upgrade
  36. 36. Philosophy • “Unix philosophy by Mike Gancarz” filtered: Make each program do one thing well. Use software leverage to your advantage. Use shell scripts to increase leverage and portability. (btw. Store data in flat text files) https://en.wikipedia.org/wiki/Unix_philosophy
  37. 37. Thanks for the patience! Questions? ! robert.lujo@gmail.com @trebor74hr

×