Stupid Buildout Tricks

Stupid Buildout Tricks
Ricardo Newbery

Wednesday, November 9, 2011

First, a basic buildout
[buildout]
parts =
pyramid
myapp

[pyramid]
recipe = zc.recipe.egg
dependent-scripts = true
eggs = pyramid
interpreter = py

[myapp]
recipe = collective.recipe.template
input = templates/myapp.ini.in
output = ${buildout:directory}/etc/myapp.ini
key_a = value_a
key_b = value_b


The problem
Multiple environments (dev, qa, staging, production) requiring
different buildout conﬁgurations.
$ bin/buildout -c dev.cfg

$ bin/buildout -c qa1.cfg

$ bin/buildout -c qa2.cfg

ad nauseum ...

The non-dev builds are performed by release engineers who
have better things to do than to read complicated release
notes to ﬁgure out the right buildout incantation for the
current environment.


The solution

Add environment variables into each non-dev environment
and have buildout ﬁgure out which conﬁg is the right one.


First try...
Used a buildout extension to munge the buildout config
dynamically. Similar to the way collective.recipe.python works.
Two issues:
1) The variant configs are hard to read.

2) Doesn't play well with mr.developer since "bin/develop"
turns off extensions when reading the config.

This was painful so I redid this w/o the buildout extensions...


Nothing fancy...
$ bin/buildout
-c config/variant-staging.cfg
buildout:directory=<<PATH TO BUILDOUT ROOT>>

The "buildout:directory" value ﬁxes the problem where
buildout assumes the build context is the directory where the
main conﬁg resides.
Now, I just need a script to generate this call to buildout...

$ bin/environ

Let's make this script...


import os
import sys
import logging
import subprocess

logger = logging.getLogger('buildout.variants')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(message)s'))
logger.propagate = False
logger.addHandler(handler)
log = logger.info

# Is config file specified on command line?
config_file = False
args_in = sys.argv[1:]
while args_in:
op = args_in.pop(0)
if op[:2] == '-c':
config_file = True
log("buildout.variants: Disabled.")

# If config file not specified, check for one matching the environment
args = ['bin/buildout']
if not config_file:
environ = os.environ.get('BUILDOUT_ENVIRON')
if environ:
log("buildout.variants: Detected environment '%s'" %environ)
config_file = 'config/variant-%s.cfg' %environ
if not os.path.exists(config_file):
log("buildout.variants: Not found '%s'" %config_file)
config_file = False
if config_file:
args.append('-c%s' %config_file)
args.append('buildout:directory=%s' %os.getcwd())
log("buildout.variants: Applying variant '%s'" %config_file)
else:
log("buildout.variants: No variants found.")

# Run buildout
args = args + sys.argv[1:]
subprocess.call(args)


# Is config file specified on command line?

config_file = False
args_in = sys.argv[1:]
while args_in:
op = args_in.pop(0)
if op[:2] == '-c':
config_file = True
log("buildout.variants: Disabled.")


# Check the environment

args = ['bin/buildout']

if not config_file:
environ = os.environ.get('BUILDOUT_ENVIRON')
if environ:
log("buildout.variants: Detected '%s'" %environ)
config_file = 'config/variant-%s.cfg' %environ
if not os.path.exists(config_file):
log("buildout.variants: Not found '%s'" %config_file)
config_file = False
if config_file:
args.append('-c%s' %config_file)
args.append('buildout:directory=%s' %os.getcwd())
log("buildout.variants: Applying variant '%s'" %config_file)
else:
log("buildout.variants: No variants found.")


# Run buildout script

args = args + sys.argv[1:]
subprocess.call(args)


release engineers = :-)
Only need to know one command to update a buildout
$ bin/environ

Environment buildouts can always be overridden.
$ bin/environ -c local.cfg

or
$ BUILDOUT_ENVIRON=staging bin/environ


Extra credit
All the documentation says to run buildout with...
$ bin/buildout

Let's not confuse the release engineers with yet another way
to run a buildout so I just created a custom buildout script
(generated by the bootstrap.py) that folds this environment
hook into bin/buildout itself.
[customizing buildout script left as an exercise for the reader]
So no more 'bin/environ'. Just 'bin/buildout' again.
Release engineers even happier :-)


An alternative
Another way to get environment-aware buildout configurations
is with mr.scripty,

http://pypi.python.org/pypi/mr.scripty

Allows defining values in a buildout part with regular python
expressions. This is possibly a more general solution although
the configs may be a little harder to read. You also can't adjust
a buildout config "extends" value with this method.


Another problem
Staging environment sucks.
Occasionally, stuff in python site packages in staging borks the
buildout. I could work on ﬁxing the staging environment but
I've got better things to do.
So let's isolate the buildout from site packages.
Some possible solutions...

1) virtualenv

2) zc.buildout, version 1.5.x

Let's show the virtualenv method...


Release notes 2.0

Using virtualenv...

$ hg clone <path-to-repo> MyBuildout

$ virtualenv --no-site-packages MyBuildout

$ cd MyBuildout

$ bin/python bootstrap.py

$ bin/buildout


A virtualenv bootstrap
Virtualenv must be installed, or we need to create a bootstrap.

$ virtualenv.create_bootstrap_script(extra_text)

The extra_text is just some python script to hook in extra
behavior during a bootstrap.
extend_parser(optparse_parser):
Add or remove options from the parser here.

adjust_options(options, args):
Change options or args here.

after_install(options, home_dir):
After everything is installed, this function is called.

See the virtualenv docs for details.


(cont.)
>>> extra_text = """
>>> def adjust_options(options, args):
>>>
>>> # Create virtualenv in current dir
>>> args = ['.']
>>>
>>> # Isolate from system python site-packages
>>> options.no_site_packages = True
>>>
>>> # Use distribute instead of setuptools
>>> options.use_distribute = True
>>> """
>>>
>>> import virtualenv
>>> out = virtualenv.create_bootstrap_script(extra_text)
>>> open('virtualenv.py', 'w').write(out)


Release notes 3.0
Virtualenv not installed in system python...


$ cd MyBuildout

$ python2.6 virtualenv.py

$ bin/python bootstrap.py

$ bin/buildout


But wait, there's more

Add the following to the virtualenv bootstrap script,

def after_install(options, home_dir):
if os.path.exists('bootstrap.py'):
logger.notify('Running bootstrap.py')
subprocess.call([os.path.join(home_dir, 'bin', 'python'),
'bootstrap.py'])

Now bootstrapping virtualenv also bootstraps buildout.


Release notes 4.0


$ cd MyBuildout

$ python2.6 virtualenv.py

$ bin/buildout


Conclusion

* Much simpler release process... happy release engineers.

* Much simpler release notes... happy developers

* Joy all around.


Stupid Buildout Tricks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Stupid Buildout Tricks

Similar to Stupid Buildout Tricks (20)

Recently uploaded

Recently uploaded (20)

Stupid Buildout Tricks