History <ul><li>Made by Jim Fulton. 2006
“It should be possible to check-in a buildout specification and reproduce the same software” </li></ul>
What problem does it solve?
Isolating Python Libraries Like virtualenv
Managing depencies/versions <ul><li>Bit like pip requires.txt  </li></ul>
Compiling/Building stuff Bit like Make or Ant
Installing/Configuring/Deploying Bit like Chef or Puppet
Checkout Development Code Bit like svn externals
Simplifying Config Templates  Bit like Paster Script
But wait there's more! <ul><li>We'll do all that in one .cfg file </li></ul>
So what is buildout? Gaffer tape is like the force. It has a light side, a dark  side, and it holds the universe together.
App Stacks <ul><li>Nginix, Varinish, HAProxy, Zope, Plone, MySQL
Perl, Munin
Pyramid, Gevent, pyramid_socketio, Redis
Django, customapp1, customapp2, customapp3
Etc etc </li></ul>
Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1
Initialising $ easy_install zc.buildout … Finished processing dependencies for zc.buildout $ buildout init Creating direct...
“What goes on in buildout, stays in buildout”
Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-d...
Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buil...
Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillo...
Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow #  if packag...
Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [...
Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','w...
mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git ...
zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldP...
Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs =  Plone >= 4.1
Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plo...
Versions KGS [ buildout] extends =  http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] re...
Version Conflicts Installing. Getting section zeoserver. Initializing part zeoserver. Error: There is a version conflict. ...
Version Conflict Prevention $  bin/buildout -N (prevent auto upgrading packages) $  bin/buidout -v  (discover where confli...
dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.d...
Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg
Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone h...
mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ...  ...
mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlo...
Extending #staging.cfg [buildout] extends = buildout.cfg [ports] OFFSET=8000
Upcoming SlideShare
Loading in...5
×

Buildout: How to maintain big app stacks without losing your mind

2,283

Published on

Slides used in pyconau 2011 talk on zc.buildout talk.

Published in: Technology, Self Improvement
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,283
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
23
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Buildout: How to maintain big app stacks without losing your mind"

  1. 1. Buildout How to maintain big app stacks without losing your mind Dylan Jay Sypy.org
  2. 2. History <ul><li>Made by Jim Fulton. 2006
  3. 3. “It should be possible to check-in a buildout specification and reproduce the same software” </li></ul>
  4. 4. What problem does it solve?
  5. 5. Isolating Python Libraries Like virtualenv
  6. 6. Managing depencies/versions <ul><li>Bit like pip requires.txt </li></ul>
  7. 7. Compiling/Building stuff Bit like Make or Ant
  8. 8. Installing/Configuring/Deploying Bit like Chef or Puppet
  9. 9. Checkout Development Code Bit like svn externals
  10. 10. Simplifying Config Templates Bit like Paster Script
  11. 11. But wait there's more! <ul><li>We'll do all that in one .cfg file </li></ul>
  12. 12. So what is buildout? Gaffer tape is like the force. It has a light side, a dark side, and it holds the universe together.
  13. 13. App Stacks <ul><li>Nginix, Varinish, HAProxy, Zope, Plone, MySQL
  14. 14. Perl, Munin
  15. 15. Pyramid, Gevent, pyramid_socketio, Redis
  16. 16. Django, customapp1, customapp2, customapp3
  17. 17. Etc etc </li></ul>
  18. 18. Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1
  19. 19. Initialising $ easy_install zc.buildout … Finished processing dependencies for zc.buildout $ buildout init Creating directory '/Users/dylanjay/Projects/sandpit/buildout/bin'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/parts'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/develop-eggs'. Generated script '/Users/dylanjay/Projects/sandpit/buildout/bin/buildout'. $ bin/buildout
  20. 20. “What goes on in buildout, stays in buildout”
  21. 21. Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  22. 22. Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [ chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  23. 23. Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillow Interpreter = python $ bin/python -m fabulous.text hello world
  24. 24. Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow # if package has its own entry-point, you don't need the stuff below initialization = from fabulous import text; print text.Text(&quot;Hello World&quot;, shadow=True) go = lambda: True scripts = hello entry-points = hello=__main__:go
  25. 25. Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [ '/../download-cache/eggs/elementtree-1.2.7_20070827_preview-py2.4.egg', '/../download-cache/eggs/archetypes.kss-1.4.3-py2.4.egg', … ] import plone.recipe.zope2instance.ctl if __name__ == '__main__': plone.recipe.zope2instance.ctl.main( [&quot;-C&quot;, '/Users/dylanjay/Projects/gcio/parts/instance/etc/zope.conf'] + sys.argv[1:])
  26. 26. Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','world') self.output = options['output'] = 'hello %s' % self.input self.location = options['location'] = self.buildout['buildout']['bin-directory']+'hello' def install(self): # Return files that were created by the recipe. The buildout # will remove all returned files upon reinstall. f = open(self.location).write(self.output) ; f.close() return [self.location] def update(self): pass
  27. 27. mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git git://github.com/topher200/genetic-hello-world-python.git [ helloworld] recipe=zc.recipe.egg eggs = genetic-hello-world-python # WARNING: this example doesn't work
  28. 28. zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone Http-address = 127.0.0.1:8080 [varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:8080 backends = ${instance:http-address}
  29. 29. Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone >= 4.1
  30. 30. Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plone= 4.1
  31. 31. Versions KGS [ buildout] extends = http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone
  32. 32. Version Conflicts Installing. Getting section zeoserver. Initializing part zeoserver. Error: There is a version conflict. We already have: zope.component 3.8.0 but five.localsitemanager 1.1 requires 'zope.component<3.6dev'. Conflicts are caused by buildout having no way to know depenency specs until its too late.
  33. 33. Version Conflict Prevention $ bin/buildout -N (prevent auto upgrading packages) $ bin/buidout -v (discover where conflict arose) Or Pinning versions
  34. 34. dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.dumppickedversions'. ... *************** PICKED VERSIONS **************** [versions] myegg = 1.1 setuptools = 2.1 zc.buildout = 1.5.3 zc.recipe.egg = 1.5.2 <BLANKLINE> *************** /PICKED VERSIONS ***************
  35. 35. Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg
  36. 36. Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = 127.0.0.1:8080 [instance2] <=instance1 http-address = 127.0.0.1:8081 [ instance2] <= instance1 http-address = 127.0.0.1:8081 [ varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [ varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:80 backends = ${ instance:http-address} ${ instance1:http-address} ${ instance2:http-address}
  37. 37. mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ... for key,value in self.buildout['ports_base'].items(): ... self.options[key] = str(int(value)+int(self.OFFSET))
  38. 38. mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = ${ports:instance1} [ instance2] <= instance1 http-address = ${ports:instance2} [ instance2] <= instance1 http-address = ${ports:instance3}
  39. 39. Extending #staging.cfg [buildout] extends = buildout.cfg [ports] OFFSET=8000
  40. 40. Annotation mode $ bin/buildout annotate Annotated sections ================== [bfg] dependent-scripts= true /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg eggs= repoze.bfg mobme /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg index= http://dist.repoze.org/bfg/current/simple /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg recipe= zc.recipe.egg /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg [buildout] accept-buildout-test-releases= false DEFAULT_VALUE
  41. 41. Other Recipes
  42. 42. collective.hostout <ul><li>Deploys a buildout to a new location (host)
  43. 43. Uses Fabric under the hood </li></ul>
  44. 44. Thanks <ul><li>http://www.buildout.org
  45. 45. http://pypi.python.org/pypi?%3Aaction=search&term=recipe
  46. 46. Dylan Jay
  47. 47. Twitter: djay75
  48. 48. http://www.pretaweb.com </li></ul>

×