Your SlideShare is downloading. ×
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Buildout: How to maintain big app stacks without losing your mind

2,181
views

Published on

Slides used in pyconau 2011 talk on zc.buildout talk.

Slides used in pyconau 2011 talk on zc.buildout talk.

Published in: Technology, Self Improvement

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,181
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
23
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Buildout How to maintain big app stacks without losing your mind Dylan Jay Sypy.org
  • 2. History
    • Made by Jim Fulton. 2006
    • 3. “It should be possible to check-in a buildout specification and reproduce the same software”
  • 4. What problem does it solve?
  • 5. Isolating Python Libraries Like virtualenv
  • 6. Managing depencies/versions
    • Bit like pip requires.txt
  • 7. Compiling/Building stuff Bit like Make or Ant
  • 8. Installing/Configuring/Deploying Bit like Chef or Puppet
  • 9. Checkout Development Code Bit like svn externals
  • 10. Simplifying Config Templates Bit like Paster Script
  • 11. But wait there's more!
    • We'll do all that in one .cfg file
  • 12. So what is buildout? Gaffer tape is like the force. It has a light side, a dark side, and it holds the universe together.
  • 13. App Stacks
    • Nginix, Varinish, HAProxy, Zope, Plone, MySQL
    • 14. Perl, Munin
    • 15. Pyramid, Gevent, pyramid_socketio, Redis
    • 16. Django, customapp1, customapp2, customapp3
    • 17. Etc etc
  • 18. Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1
  • 19. Initialising $ easy_install zc.buildout … Finished processing dependencies for zc.buildout $ buildout init Creating directory '/Users/dylanjay/Projects/sandpit/buildout/bin'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/parts'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/develop-eggs'. Generated script '/Users/dylanjay/Projects/sandpit/buildout/bin/buildout'. $ bin/buildout
  • 20. “What goes on in buildout, stays in buildout”
  • 21. Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  • 22. Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [ chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  • 23. Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillow Interpreter = python $ bin/python -m fabulous.text hello world
  • 24. Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow # if package has its own entry-point, you don't need the stuff below initialization = from fabulous import text; print text.Text("Hello World", shadow=True) go = lambda: True scripts = hello entry-points = hello=__main__:go
  • 25. Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [ '/../download-cache/eggs/elementtree-1.2.7_20070827_preview-py2.4.egg', '/../download-cache/eggs/archetypes.kss-1.4.3-py2.4.egg', … ] import plone.recipe.zope2instance.ctl if __name__ == '__main__': plone.recipe.zope2instance.ctl.main( ["-C", '/Users/dylanjay/Projects/gcio/parts/instance/etc/zope.conf'] + sys.argv[1:])
  • 26. Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','world') self.output = options['output'] = 'hello %s' % self.input self.location = options['location'] = self.buildout['buildout']['bin-directory']+'hello' def install(self): # Return files that were created by the recipe. The buildout # will remove all returned files upon reinstall. f = open(self.location).write(self.output) ; f.close() return [self.location] def update(self): pass
  • 27. mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git git://github.com/topher200/genetic-hello-world-python.git [ helloworld] recipe=zc.recipe.egg eggs = genetic-hello-world-python # WARNING: this example doesn't work
  • 28. zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone Http-address = 127.0.0.1:8080 [varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:8080 backends = ${instance:http-address}
  • 29. Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone >= 4.1
  • 30. Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plone= 4.1
  • 31. Versions KGS [ buildout] extends = http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone
  • 32. Version Conflicts Installing. Getting section zeoserver. Initializing part zeoserver. Error: There is a version conflict. We already have: zope.component 3.8.0 but five.localsitemanager 1.1 requires 'zope.component<3.6dev'. Conflicts are caused by buildout having no way to know depenency specs until its too late.
  • 33. Version Conflict Prevention $ bin/buildout -N (prevent auto upgrading packages) $ bin/buidout -v (discover where conflict arose) Or Pinning versions
  • 34. dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.dumppickedversions'. ... *************** PICKED VERSIONS **************** [versions] myegg = 1.1 setuptools = 2.1 zc.buildout = 1.5.3 zc.recipe.egg = 1.5.2 <BLANKLINE> *************** /PICKED VERSIONS ***************
  • 35. Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg
  • 36. Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = 127.0.0.1:8080 [instance2] <=instance1 http-address = 127.0.0.1:8081 [ instance2] <= instance1 http-address = 127.0.0.1:8081 [ varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [ varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:80 backends = ${ instance:http-address} ${ instance1:http-address} ${ instance2:http-address}
  • 37. mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ... for key,value in self.buildout['ports_base'].items(): ... self.options[key] = str(int(value)+int(self.OFFSET))
  • 38. mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = ${ports:instance1} [ instance2] <= instance1 http-address = ${ports:instance2} [ instance2] <= instance1 http-address = ${ports:instance3}
  • 39. Extending #staging.cfg [buildout] extends = buildout.cfg [ports] OFFSET=8000
  • 40. Annotation mode $ bin/buildout annotate Annotated sections ================== [bfg] dependent-scripts= true /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg eggs= repoze.bfg mobme /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg index= http://dist.repoze.org/bfg/current/simple /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg recipe= zc.recipe.egg /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg [buildout] accept-buildout-test-releases= false DEFAULT_VALUE
  • 41. Other Recipes
  • 42. collective.hostout
    • Deploys a buildout to a new location (host)
    • 43. Uses Fabric under the hood
  • 44. Thanks
    • http://www.buildout.org
    • 45. http://pypi.python.org/pypi?%3Aaction=search&term=recipe
    • 46. Dylan Jay
    • 47. Twitter: djay75
    • 48. http://www.pretaweb.com