Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Buildout: How to maintain big app stacks without losing your mind

on

  • 2,583 views

Slides used in pyconau 2011 talk on zc.buildout talk.

Slides used in pyconau 2011 talk on zc.buildout talk.

Statistics

Views

Total Views
2,583
Views on SlideShare
2,545
Embed Views
38

Actions

Likes
3
Downloads
22
Comments
0

4 Embeds 38

http://lanyrd.com 21
http://www.python.rk.edu.pl 9
http://tweetedtimes.com 6
http://twitter.com 2

Accessibility

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Buildout: How to maintain big app stacks without losing your mind Presentation Transcript

  • 1. Buildout How to maintain big app stacks without losing your mind Dylan Jay Sypy.org
  • 2. History
    • Made by Jim Fulton. 2006
    • 3. “It should be possible to check-in a buildout specification and reproduce the same software”
  • 4. What problem does it solve?
  • 5. Isolating Python Libraries Like virtualenv
  • 6. Managing depencies/versions
    • Bit like pip requires.txt
  • 7. Compiling/Building stuff Bit like Make or Ant
  • 8. Installing/Configuring/Deploying Bit like Chef or Puppet
  • 9. Checkout Development Code Bit like svn externals
  • 10. Simplifying Config Templates Bit like Paster Script
  • 11. But wait there's more!
    • We'll do all that in one .cfg file
  • 12. So what is buildout? Gaffer tape is like the force. It has a light side, a dark side, and it holds the universe together.
  • 13. App Stacks
    • Nginix, Varinish, HAProxy, Zope, Plone, MySQL
    • 14. Perl, Munin
    • 15. Pyramid, Gevent, pyramid_socketio, Redis
    • 16. Django, customapp1, customapp2, customapp3
    • 17. Etc etc
  • 18. Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1
  • 19. Initialising $ easy_install zc.buildout … Finished processing dependencies for zc.buildout $ buildout init Creating directory '/Users/dylanjay/Projects/sandpit/buildout/bin'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/parts'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/develop-eggs'. Generated script '/Users/dylanjay/Projects/sandpit/buildout/bin/buildout'. $ bin/buildout
  • 20. “What goes on in buildout, stays in buildout”
  • 21. Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  • 22. Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [ chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
  • 23. Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillow Interpreter = python $ bin/python -m fabulous.text hello world
  • 24. Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow # if package has its own entry-point, you don't need the stuff below initialization = from fabulous import text; print text.Text("Hello World", shadow=True) go = lambda: True scripts = hello entry-points = hello=__main__:go
  • 25. Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [ '/../download-cache/eggs/elementtree-1.2.7_20070827_preview-py2.4.egg', '/../download-cache/eggs/archetypes.kss-1.4.3-py2.4.egg', … ] import plone.recipe.zope2instance.ctl if __name__ == '__main__': plone.recipe.zope2instance.ctl.main( ["-C", '/Users/dylanjay/Projects/gcio/parts/instance/etc/zope.conf'] + sys.argv[1:])
  • 26. Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','world') self.output = options['output'] = 'hello %s' % self.input self.location = options['location'] = self.buildout['buildout']['bin-directory']+'hello' def install(self): # Return files that were created by the recipe. The buildout # will remove all returned files upon reinstall. f = open(self.location).write(self.output) ; f.close() return [self.location] def update(self): pass
  • 27. mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git git://github.com/topher200/genetic-hello-world-python.git [ helloworld] recipe=zc.recipe.egg eggs = genetic-hello-world-python # WARNING: this example doesn't work
  • 28. zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone Http-address = 127.0.0.1:8080 [varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:8080 backends = ${instance:http-address}
  • 29. Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone >= 4.1
  • 30. Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plone= 4.1
  • 31. Versions KGS [ buildout] extends = http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone
  • 32. Version Conflicts Installing. Getting section zeoserver. Initializing part zeoserver. Error: There is a version conflict. We already have: zope.component 3.8.0 but five.localsitemanager 1.1 requires 'zope.component<3.6dev'. Conflicts are caused by buildout having no way to know depenency specs until its too late.
  • 33. Version Conflict Prevention $ bin/buildout -N (prevent auto upgrading packages) $ bin/buidout -v (discover where conflict arose) Or Pinning versions
  • 34. dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.dumppickedversions'. ... *************** PICKED VERSIONS **************** [versions] myegg = 1.1 setuptools = 2.1 zc.buildout = 1.5.3 zc.recipe.egg = 1.5.2 <BLANKLINE> *************** /PICKED VERSIONS ***************
  • 35. Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg
  • 36. Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = 127.0.0.1:8080 [instance2] <=instance1 http-address = 127.0.0.1:8081 [ instance2] <= instance1 http-address = 127.0.0.1:8081 [ varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [ varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:80 backends = ${ instance:http-address} ${ instance1:http-address} ${ instance2:http-address}
  • 37. mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ... for key,value in self.buildout['ports_base'].items(): ... self.options[key] = str(int(value)+int(self.OFFSET))
  • 38. mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = ${ports:instance1} [ instance2] <= instance1 http-address = ${ports:instance2} [ instance2] <= instance1 http-address = ${ports:instance3}
  • 39. Extending #staging.cfg [buildout] extends = buildout.cfg [ports] OFFSET=8000
  • 40. Annotation mode $ bin/buildout annotate Annotated sections ================== [bfg] dependent-scripts= true /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg eggs= repoze.bfg mobme /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg index= http://dist.repoze.org/bfg/current/simple /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg recipe= zc.recipe.egg /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg [buildout] accept-buildout-test-releases= false DEFAULT_VALUE
  • 41. Other Recipes
  • 42. collective.hostout
    • Deploys a buildout to a new location (host)
    • 43. Uses Fabric under the hood
  • 44. Thanks
    • http://www.buildout.org
    • 45. http://pypi.python.org/pypi?%3Aaction=search&term=recipe
    • 46. Dylan Jay
    • 47. Twitter: djay75
    • 48. http://www.pretaweb.com