Stupid Buildout Tricks

  • 2,740 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
2,740
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
9
Comments
1
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Stupid Buildout Tricks Ricardo NewberyWednesday, November 9, 2011
  • 2. First, a basic buildout [buildout] parts = pyramid myapp [pyramid] recipe = zc.recipe.egg dependent-scripts = true eggs = pyramid interpreter = py [myapp] recipe = collective.recipe.template input = templates/myapp.ini.in output = ${buildout:directory}/etc/myapp.ini key_a = value_a key_b = value_bWednesday, November 9, 2011
  • 3. The problem Multiple environments (dev, qa, staging, production) requiring different buildout configurations. $ bin/buildout -c dev.cfg $ bin/buildout -c qa1.cfg $ bin/buildout -c qa2.cfg ad nauseum ... The non-dev builds are performed by release engineers who have better things to do than to read complicated release notes to figure out the right buildout incantation for the current environment.Wednesday, November 9, 2011
  • 4. The solution Add environment variables into each non-dev environment and have buildout figure out which config is the right one.Wednesday, November 9, 2011
  • 5. First try... Used a buildout extension to munge the buildout config dynamically. Similar to the way collective.recipe.python works. Two issues: 1) The variant configs are hard to read. 2) Doesnt play well with mr.developer since "bin/develop" turns off extensions when reading the config. This was painful so I redid this w/o the buildout extensions...Wednesday, November 9, 2011
  • 6. Nothing fancy... $ bin/buildout -c config/variant-staging.cfg buildout:directory=<<PATH TO BUILDOUT ROOT>> The "buildout:directory" value fixes the problem where buildout assumes the build context is the directory where the main config resides. Now, I just need a script to generate this call to buildout... $ bin/environ Lets make this script...Wednesday, November 9, 2011
  • 7. import os import sys import logging import subprocess logger = logging.getLogger(buildout.variants) logger.setLevel(logging.INFO) handler = logging.StreamHandler() handler.setFormatter(logging.Formatter(%(message)s)) logger.propagate = False logger.addHandler(handler) log = logger.info # Is config file specified on command line? config_file = False args_in = sys.argv[1:] while args_in: op = args_in.pop(0) if op[:2] == -c: config_file = True log("buildout.variants: Disabled.") # If config file not specified, check for one matching the environment args = [bin/buildout] if not config_file: environ = os.environ.get(BUILDOUT_ENVIRON) if environ: log("buildout.variants: Detected environment %s" %environ) config_file = config/variant-%s.cfg %environ if not os.path.exists(config_file): log("buildout.variants: Not found %s" %config_file) config_file = False if config_file: args.append(-c%s %config_file) args.append(buildout:directory=%s %os.getcwd()) log("buildout.variants: Applying variant %s" %config_file) else: log("buildout.variants: No variants found.") # Run buildout args = args + sys.argv[1:] subprocess.call(args)Wednesday, November 9, 2011
  • 8. # Is config file specified on command line? config_file = False args_in = sys.argv[1:] while args_in: op = args_in.pop(0) if op[:2] == -c: config_file = True log("buildout.variants: Disabled.")Wednesday, November 9, 2011
  • 9. # Check the environment args = [bin/buildout] if not config_file: environ = os.environ.get(BUILDOUT_ENVIRON) if environ: log("buildout.variants: Detected %s" %environ) config_file = config/variant-%s.cfg %environ if not os.path.exists(config_file): log("buildout.variants: Not found %s" %config_file) config_file = False if config_file: args.append(-c%s %config_file) args.append(buildout:directory=%s %os.getcwd()) log("buildout.variants: Applying variant %s" %config_file) else: log("buildout.variants: No variants found.")Wednesday, November 9, 2011
  • 10. # Run buildout script args = args + sys.argv[1:] subprocess.call(args)Wednesday, November 9, 2011
  • 11. release engineers = :-) Only need to know one command to update a buildout $ bin/environ Environment buildouts can always be overridden. $ bin/environ -c local.cfg or $ BUILDOUT_ENVIRON=staging bin/environWednesday, November 9, 2011
  • 12. Extra credit All the documentation says to run buildout with... $ bin/buildout Lets not confuse the release engineers with yet another way to run a buildout so I just created a custom buildout script (generated by the bootstrap.py) that folds this environment hook into bin/buildout itself. [customizing buildout script left as an exercise for the reader] So no more bin/environ. Just bin/buildout again. Release engineers even happier :-)Wednesday, November 9, 2011
  • 13. An alternative Another way to get environment-aware buildout configurations is with mr.scripty, http://pypi.python.org/pypi/mr.scripty Allows defining values in a buildout part with regular python expressions. This is possibly a more general solution although the configs may be a little harder to read. You also cant adjust a buildout config "extends" value with this method.Wednesday, November 9, 2011
  • 14. Another problem Staging environment sucks. Occasionally, stuff in python site packages in staging borks the buildout. I could work on fixing the staging environment but Ive got better things to do. So lets isolate the buildout from site packages. Some possible solutions... 1) virtualenv 2) zc.buildout, version 1.5.x Lets show the virtualenv method...Wednesday, November 9, 2011
  • 15. Release notes 2.0 Using virtualenv... $ hg clone <path-to-repo> MyBuildout $ virtualenv --no-site-packages MyBuildout $ cd MyBuildout $ bin/python bootstrap.py $ bin/buildoutWednesday, November 9, 2011
  • 16. A virtualenv bootstrap Virtualenv must be installed, or we need to create a bootstrap. $ virtualenv.create_bootstrap_script(extra_text) The extra_text is just some python script to hook in extra behavior during a bootstrap. extend_parser(optparse_parser): Add or remove options from the parser here. adjust_options(options, args): Change options or args here. after_install(options, home_dir): After everything is installed, this function is called. See the virtualenv docs for details.Wednesday, November 9, 2011
  • 17. (cont.) >>> extra_text = """ >>> def adjust_options(options, args): >>> >>> # Create virtualenv in current dir >>> args = [.] >>> >>> # Isolate from system python site-packages >>> options.no_site_packages = True >>> >>> # Use distribute instead of setuptools >>> options.use_distribute = True >>> """ >>> >>> import virtualenv >>> out = virtualenv.create_bootstrap_script(extra_text) >>> open(virtualenv.py, w).write(out)Wednesday, November 9, 2011
  • 18. Release notes 3.0 Virtualenv not installed in system python... $ hg clone <path-to-repo> MyBuildout $ cd MyBuildout $ python2.6 virtualenv.py $ bin/python bootstrap.py $ bin/buildoutWednesday, November 9, 2011
  • 19. But wait, theres more Add the following to the virtualenv bootstrap script, def after_install(options, home_dir): if os.path.exists(bootstrap.py): logger.notify(Running bootstrap.py) subprocess.call([os.path.join(home_dir, bin, python), bootstrap.py]) Now bootstrapping virtualenv also bootstraps buildout.Wednesday, November 9, 2011
  • 20. Release notes 4.0 $ hg clone <path-to-repo> MyBuildout $ cd MyBuildout $ python2.6 virtualenv.py $ bin/buildoutWednesday, November 9, 2011
  • 21. Conclusion * Much simpler release process... happy release engineers. * Much simpler release notes... happy developers * Joy all around.Wednesday, November 9, 2011