Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Buildout How to maintain big app stacks without losing your mind Dylan Jay Sypy.org
History <ul><li>Made by Jim Fulton. 2006
“It should be possible to check-in a buildout specification and reproduce the same software” </li></ul>
What problem does it solve?
Isolating Python Libraries Like virtualenv
Managing depencies/versions <ul><li>Bit like pip requires.txt  </li></ul>
Compiling/Building stuff Bit like Make or Ant
Installing/Configuring/Deploying Bit like Chef or Puppet
Checkout Development Code Bit like svn externals
Simplifying Config Templates  Bit like Paster Script
But wait there's more! <ul><li>We'll do all that in one .cfg file </li></ul>
So what is buildout? Gaffer tape is like the force. It has a light side, a dark  side, and it holds the universe together.
App Stacks <ul><li>Nginix, Varinish, HAProxy, Zope, Plone, MySQL
Perl, Munin
Pyramid, Gevent, pyramid_socketio, Redis
Django, customapp1, customapp2, customapp3
Etc etc </li></ul>
Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1
Initialising $ easy_install zc.buildout … Finished processing dependencies for zc.buildout $ buildout init Creating direct...
“What goes on in buildout, stays in buildout”
Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-d...
Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buil...
Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillo...
Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow #  if packag...
Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [...
Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','w...
mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git ...
zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldP...
Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs =  Plone >= 4.1
Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plo...
Versions KGS [ buildout] extends =  http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] re...
Version Conflicts Installing. Getting section zeoserver. Initializing part zeoserver. Error: There is a version conflict. ...
Version Conflict Prevention $  bin/buildout -N (prevent auto upgrading packages) $  bin/buidout -v  (discover where confli...
dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.d...
Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg
Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone h...
mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ...  ...
mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlo...
Upcoming SlideShare
Loading in …5
×

Buildout: How to maintain big app stacks without losing your mind

Slides used in pyconau 2011 talk on zc.buildout talk.

  • Be the first to comment

Buildout: How to maintain big app stacks without losing your mind

  1. 1. Buildout How to maintain big app stacks without losing your mind Dylan Jay Sypy.org
  2. 2. History <ul><li>Made by Jim Fulton. 2006
  3. 3. “It should be possible to check-in a buildout specification and reproduce the same software” </li></ul>
  4. 4. What problem does it solve?
  5. 5. Isolating Python Libraries Like virtualenv
  6. 6. Managing depencies/versions <ul><li>Bit like pip requires.txt </li></ul>
  7. 7. Compiling/Building stuff Bit like Make or Ant
  8. 8. Installing/Configuring/Deploying Bit like Chef or Puppet
  9. 9. Checkout Development Code Bit like svn externals
  10. 10. Simplifying Config Templates Bit like Paster Script
  11. 11. But wait there's more! <ul><li>We'll do all that in one .cfg file </li></ul>
  12. 12. So what is buildout? Gaffer tape is like the force. It has a light side, a dark side, and it holds the universe together.
  13. 13. App Stacks <ul><li>Nginix, Varinish, HAProxy, Zope, Plone, MySQL
  14. 14. Perl, Munin
  15. 15. Pyramid, Gevent, pyramid_socketio, Redis
  16. 16. Django, customapp1, customapp2, customapp3
  17. 17. Etc etc </li></ul>
  18. 18. Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1
  19. 19. Initialising $ easy_install zc.buildout … Finished processing dependencies for zc.buildout $ buildout init Creating directory '/Users/dylanjay/Projects/sandpit/buildout/bin'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/parts'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/develop-eggs'. Generated script '/Users/dylanjay/Projects/sandpit/buildout/bin/buildout'. $ bin/buildout
  20. 20. “What goes on in buildout, stays in buildout”
  21. 21. Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  22. 22. Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [ chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  23. 23. Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillow Interpreter = python $ bin/python -m fabulous.text hello world
  24. 24. Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow # if package has its own entry-point, you don't need the stuff below initialization = from fabulous import text; print text.Text(&quot;Hello World&quot;, shadow=True) go = lambda: True scripts = hello entry-points = hello=__main__:go
  25. 25. Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [ '/../download-cache/eggs/elementtree-1.2.7_20070827_preview-py2.4.egg', '/../download-cache/eggs/archetypes.kss-1.4.3-py2.4.egg', … ] import plone.recipe.zope2instance.ctl if __name__ == '__main__': plone.recipe.zope2instance.ctl.main( [&quot;-C&quot;, '/Users/dylanjay/Projects/gcio/parts/instance/etc/zope.conf'] + sys.argv[1:])
  26. 26. Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','world') self.output = options['output'] = 'hello %s' % self.input self.location = options['location'] = self.buildout['buildout']['bin-directory']+'hello' def install(self): # Return files that were created by the recipe. The buildout # will remove all returned files upon reinstall. f = open(self.location).write(self.output) ; f.close() return [self.location] def update(self): pass
  27. 27. mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git git://github.com/topher200/genetic-hello-world-python.git [ helloworld] recipe=zc.recipe.egg eggs = genetic-hello-world-python # WARNING: this example doesn't work
  28. 28. zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone Http-address = 127.0.0.1:8080 [varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:8080 backends = ${instance:http-address}
  29. 29. Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone >= 4.1
  30. 30. Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plone= 4.1
  31. 31. Versions KGS [ buildout] extends = http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone
  32. 32. Version Conflicts Installing. Getting section zeoserver. Initializing part zeoserver. Error: There is a version conflict. We already have: zope.component 3.8.0 but five.localsitemanager 1.1 requires 'zope.component<3.6dev'. Conflicts are caused by buildout having no way to know depenency specs until its too late.
  33. 33. Version Conflict Prevention $ bin/buildout -N (prevent auto upgrading packages) $ bin/buidout -v (discover where conflict arose) Or Pinning versions
  34. 34. dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.dumppickedversions'. ... *************** PICKED VERSIONS **************** [versions] myegg = 1.1 setuptools = 2.1 zc.buildout = 1.5.3 zc.recipe.egg = 1.5.2 <BLANKLINE> *************** /PICKED VERSIONS ***************
  35. 35. Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg
  36. 36. Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = 127.0.0.1:8080 [instance2] <=instance1 http-address = 127.0.0.1:8081 [ instance2] <= instance1 http-address = 127.0.0.1:8081 [ varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [ varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:80 backends = ${ instance:http-address} ${ instance1:http-address} ${ instance2:http-address}
  37. 37. mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ... for key,value in self.buildout['ports_base'].items(): ... self.options[key] = str(int(value)+int(self.OFFSET))
  38. 38. mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = ${ports:instance1} [ instance2] <= instance1 http-address = ${ports:instance2} [ instance2] <= instance1 http-address = ${ports:instance3}
  39. 39. Extending #staging.cfg [buildout] extends = buildout.cfg [ports] OFFSET=8000
  40. 40. Annotation mode $ bin/buildout annotate Annotated sections ================== [bfg] dependent-scripts= true /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg eggs= repoze.bfg mobme /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg index= http://dist.repoze.org/bfg/current/simple /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg recipe= zc.recipe.egg /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg [buildout] accept-buildout-test-releases= false DEFAULT_VALUE
  41. 41. Other Recipes
  42. 42. collective.hostout <ul><li>Deploys a buildout to a new location (host)
  43. 43. Uses Fabric under the hood </li></ul>
  44. 44. Thanks <ul><li>http://www.buildout.org
  45. 45. http://pypi.python.org/pypi?%3Aaction=search&term=recipe
  46. 46. Dylan Jay
  47. 47. Twitter: djay75
  48. 48. http://www.pretaweb.com </li></ul>

×