Practical project
                          automation




donderdag 19 mei 2011
Practical project
                            automation
                   • Reinout van Rees
                   • Nelen & Schuurmans
                    • Utrecht
                    • Water consultancy
                    • Lots of python and
                        django


donderdag 19 mei 2011
Project: more than code

                   • Projects are important
                   • Deployment
                   • Colleagues
                   • Rebuilding
                   • Solution: automation

donderdag 19 mei 2011
Generic idea: automation

                   • Cobbler’s children
                   • Automate your
                        projects
                   • Less manual work
                   • Less menial work
                   • Fewer errors
                   • More of the fun work
donderdag 19 mei 2011
Build automation

                   • ./configure;make
                   • Ant, maven
                   • Buildout (python)

                   • Single command to
                        get started


donderdag 19 mei 2011
Build tool: good spot for
                       more automation

                   • Most are extensible
                   • Examples:
                    • Apache config file
                          generation
                        • Setting up cronjobs

donderdag 19 mei 2011
Apache config file:
  [apacheconf]
  recipe = collective.recipe.template
  input = ${buildout:directory}/etc/mysite.apache.conf.in
  output = ${buildout:directory}/etc/mysite.apache.conf

       <VirtualHost *:80>
         ServerName mysite.example.org
         CustomLog ${buildout:directory}/var/log/access.log combined
         ErrorLog ${buildout:directory}/var/log/error.log
         DocumentRoot ${buildout:directory}/var/www
         ...
         RewriteRule ^(.*) http://localhost:${django:gunicorn_port}$1 [P]


<VirtualHost *:80>
  ServerName mysite.example.org
  CustomLog /srv/mysite/var/log/access.log combined
  ErrorLog /srv/mysite/var/log/error.log
  DocumentRoot /srv/mysite/var/www
  ...
  RewriteRule ^(.*) http://localhost:10003$1 [P]
donderdag 19 mei 2011
Prevent missing apache
                        log dir (automatically!)
               [mkdir]
               recipe = z3c.recipe.mkdir
               paths =
                   ${buildout:directory}/generated
                   ${buildout:directory}/var/geotiffs
                   ${buildout:directory}/var/log
                   ${buildout:directory}/var/media
                   ${buildout:directory}/var/sqlite
                   ${buildout:directory}/var/static




donderdag 19 mei 2011
Automatic crontab
               [supervisor-cronjob]
               recipe = z3c.recipe.usercrontab
               times = @reboot
               command = ${buildout:bin-directory}/supervisord


               [whoosh-cronjob]
               recipe = z3c.recipe.usercrontab
               times = 13 */8 * * *
               command = ${buildout:bin-directory}/django update_index




donderdag 19 mei 2011
Test automation

                   • Jenkins/buildbot
                   • Deployment ok...
                   • ... so tests possible
                   • Prevent mistakes

donderdag 19 mei 2011
donderdag 19 mei 2011
Extra jenkins automation:
                      quality checks

                   • Code coverage
                        (‘coverage’ on pypi)
                   • Pyflakes and pep8
                        (also on pypi)
                   • Jslint (or jshint)

donderdag 19 mei 2011
donderdag 19 mei 2011
Quality: make it easy to
                      do the right thing

                   • “I really should do x,
                        but it takes too much
                        time so I’ll skip it”
                   • Releasing packages
                   • Skeleton

donderdag 19 mei 2011
donderdag 19 mei 2011
Releasing python
                             packages

                   • zest.releaser (on pypi)
                   • prerelease
                   • release
                   • postrelease

donderdag 19 mei 2011
donderdag 19 mei 2011
Practical project
                               automation
                   • Make it easier to do
                        the right thing: let
                        laziness work for you
                   • Automate your builds
                   • Automate tests
                   • Extend where
                        practical


donderdag 19 mei 2011
Questions?
            reinout.vanrees.org
            @reinoutvanrees
            reinout@vanrees.org
            www.nelen-schuurmans.nl


            photos.reinout.vanrees.org


donderdag 19 mei 2011

Practical project automation (PyGrunn conference)

  • 1.
    Practical project automation donderdag 19 mei 2011
  • 2.
    Practical project automation • Reinout van Rees • Nelen & Schuurmans • Utrecht • Water consultancy • Lots of python and django donderdag 19 mei 2011
  • 3.
    Project: more thancode • Projects are important • Deployment • Colleagues • Rebuilding • Solution: automation donderdag 19 mei 2011
  • 4.
    Generic idea: automation • Cobbler’s children • Automate your projects • Less manual work • Less menial work • Fewer errors • More of the fun work donderdag 19 mei 2011
  • 5.
    Build automation • ./configure;make • Ant, maven • Buildout (python) • Single command to get started donderdag 19 mei 2011
  • 6.
    Build tool: goodspot for more automation • Most are extensible • Examples: • Apache config file generation • Setting up cronjobs donderdag 19 mei 2011
  • 7.
    Apache config file: [apacheconf] recipe = collective.recipe.template input = ${buildout:directory}/etc/mysite.apache.conf.in output = ${buildout:directory}/etc/mysite.apache.conf <VirtualHost *:80> ServerName mysite.example.org CustomLog ${buildout:directory}/var/log/access.log combined ErrorLog ${buildout:directory}/var/log/error.log DocumentRoot ${buildout:directory}/var/www ... RewriteRule ^(.*) http://localhost:${django:gunicorn_port}$1 [P] <VirtualHost *:80> ServerName mysite.example.org CustomLog /srv/mysite/var/log/access.log combined ErrorLog /srv/mysite/var/log/error.log DocumentRoot /srv/mysite/var/www ... RewriteRule ^(.*) http://localhost:10003$1 [P] donderdag 19 mei 2011
  • 8.
    Prevent missing apache log dir (automatically!) [mkdir] recipe = z3c.recipe.mkdir paths = ${buildout:directory}/generated ${buildout:directory}/var/geotiffs ${buildout:directory}/var/log ${buildout:directory}/var/media ${buildout:directory}/var/sqlite ${buildout:directory}/var/static donderdag 19 mei 2011
  • 9.
    Automatic crontab [supervisor-cronjob] recipe = z3c.recipe.usercrontab times = @reboot command = ${buildout:bin-directory}/supervisord [whoosh-cronjob] recipe = z3c.recipe.usercrontab times = 13 */8 * * * command = ${buildout:bin-directory}/django update_index donderdag 19 mei 2011
  • 10.
    Test automation • Jenkins/buildbot • Deployment ok... • ... so tests possible • Prevent mistakes donderdag 19 mei 2011
  • 11.
  • 12.
    Extra jenkins automation: quality checks • Code coverage (‘coverage’ on pypi) • Pyflakes and pep8 (also on pypi) • Jslint (or jshint) donderdag 19 mei 2011
  • 13.
  • 14.
    Quality: make iteasy to do the right thing • “I really should do x, but it takes too much time so I’ll skip it” • Releasing packages • Skeleton donderdag 19 mei 2011
  • 15.
  • 16.
    Releasing python packages • zest.releaser (on pypi) • prerelease • release • postrelease donderdag 19 mei 2011
  • 17.
  • 18.
    Practical project automation • Make it easier to do the right thing: let laziness work for you • Automate your builds • Automate tests • Extend where practical donderdag 19 mei 2011
  • 19.
    Questions? reinout.vanrees.org @reinoutvanrees reinout@vanrees.org www.nelen-schuurmans.nl photos.reinout.vanrees.org donderdag 19 mei 2011