delivering applications
    with zc.buildout
and a distributed model
 Tarek Ziadé <tarek.ziade@ingeniweb.com>




        ...
Who am I ?
 New to Plone (~1 year) -> I worked on peripheral matters

 Used to Zope (CPS core developer)

 CTO at Ingeniwe...
It will look good on your desk
              besides Martin Aspeli’s book




Special promotion during the PloneConf :
buy...
Thanks !
  * Andreas Jung
  * Veda Williams
  * David Glick
  * Jeff Kowalczyk
  * Youenn Boussard
  * Christian Klinger
 ...
Part 1 - working with packages
Part 2 - working with zc.buildout
Part 3 - application lifecycle




                      ...
Part 1 - working with packages




                                 6
distutils
builds and distributes a package
registers and uploads it to PyPI




                                   7




 ...
get the screencast at http://ziade.org/ploneconf




                                              8




                 ...
+1



      distutils
             is
       the standard
in the whole Python community.




                             ...
-1



distutils
      is
   broken
(but there’s hope)




                      10




                           10
setuptools
simple dependencies management, namespaced package
                 egg distribution
                provides e...
“python setup.py bdist_egg”


       one single .egg file



                              12




                        ...
quot;Eggs are to Pythons as Jars are to Java...quot;



                  egg
                      =
             deploym...
“sdist” is enough, though
              (and should be mandatory)




                               14




              ...
get the screencast at http://ziade.org/ploneconf




                                             15




                 ...
easy_install my_package
                  =
 will get and install “my_package”
            from “PyPI”




               ...
+1



   setuptools
          is
the de facto standard
in the Plone community.




                          17




      ...
-1



setuptools
      is
   broken
too (pje says)

(but there’s hope)




                      18




                  ...
packaging future in Python


           ?

                             19




                                  19
PJE    Debian packagers      BSD packagers            Us !


 Ian                                              Zope 3
    ...
Packaging future in Python


hard to know.
but community-driven
come sprint with us saturday !

                          ...
It’s OK, fellows
we will eventually
make it through.




                     22




                          22
Problems with packaging ?

       #1 PyPI == SPOF

       #2 packages need privacy sometimes

       #3 plone.org/products...
Solutions ?
   #1 PyPI == SPOF
        Make a PyPI mirror

   #2 packages need privacy sometimes
        Run you own priva...
#1 Make a PyPI mirror : Smart mirroring

    easy_install collective.eggproxy




                                        ...
26




     26
In zc.buildout : the index option
[buildout]

index = http://my.mirror:8888




                 all calls will go through...
#2 Run your own private PyPI

   PloneSoftwareCenter !




                               28




                         ...
get the screencast on http://ziade.org/ploneconf




                                             29




                 ...
collective.dist
   python 2.6 new “register” and “upload” commands

   “mregister” and “mupload” in 2.4 and 2.5



   $ py...
Developer side : multiple servers with .pypirc
[distutils]
index-servers =
   pypi
   plone.org
   my-company

[pypi]
user...
Let’s try our new PyPI
  with collective.dist




                         32




                              32
get the screencast on http://ziade.org/ploneconf




                                             33




                 ...
#3 Make plone.org/products
                    PyPI compatible


       plone.org was
          suppose
to switch to that ...
Let’s try new.plone.org




                          35




                               35
get the screencast on http://ziade.org/ploneconf




                                             36




                 ...
The big picture




                  37




                       37
Yes !!!




                     38

developer #1
                          38
Yepeee !!!
        Releasing everywhere
   let’s spam PyPI and plone.org
         tchak ! tchak ! tchak !




developer #2...
Hahaha !

               We have our package all over
                       the place.


developer #3




               ...
end of part 1

questions ?


 #1 Make mirror

 #2 Run your own PyPI

 #3 push to several servers

 #3 use “mregister” and ...
Part 2 - working with zc.buildout
          (reminders)




                                    42




                   ...
5 hours in 2006




                  43




                       43
5 hours in 2006
   install python extra packages
   get zope
   install zope
   create an instance
   get extra products
 ...
Me in 2006




 45




             45
5 minutes in 2008




                    46




                         46
5 minutes in 2008
   get the “buildout”
   $ python bootstrap.py
   $ bin/buildout
   start to work




                  ...
But that was not the main reason




                            48




                                   48
2005                                Now
                                      zope.interface

                            ...
2005                               Now
                                         plone.ista

                              ...
Your     app.site
Killer
Plone    app.skin
 App
            ...




              51




                    51
zope.stuff

                   ...

  Your         plone.stuff
Distribution
                   ...

                app.si...
Python
                    site-packages



                   +100 packages
                    to run Plone



  somethi...
zc.buildout
          =
isolated environment




                       54




                            54
zc.buildout best practices
  #1 use the same layout for all your projects

  #2 make sure all developers have the same env...
#1 same layout for all projects

      project1
         docs
         buildout
         packages
         releases

     ...
easy_install collective.releaser




                                   57




                                        57
get the screencast on http://ziade.org/ploneconf




                                             58




                 ...
#2 make sure all developers have the same environment


     Warning
           Plone buildouts are source based



     W...
#3 use one cfg per target



 Typical buildout layout uses the extends feature
                   buildout.cfg
           ...
buildout.cfg :
     [buildout]
     parts =
        one
        two
                              [buildout]
             ...
demo
       creating a fresh Plone 3 buildout (Paste)
       adding the dev.cfg
       hooking a new development package
 ...
get the screencast on http://ziade.org/ploneconf




                                             63




                 ...
end of part 2

questions ?


#1 use the same layout
   for all projects

#2 make sure all developers
   have the same envi...
Part 3 - application lifecycle




                                 65
66
Releasing packages
Releasing the buildout




                         67
Releasing packages

 for package in packages:
    raise the version
    edit CHANGES.txt
    create a branch (svn)
    go ...
Releasing packages with collective.releaser:


 for package in packages:
    python setup.py release




                 ...
Defining targets in .pypirc
  [distutils]

  index-servers =
     pypi
     plone.org
     private

  glob-style = true

 ...
get the screencast on http://ziade.org/ploneconf




                                             71




                 ...
collective.releaser makes a lot of assumptions
      it’s svn-oriented (the first use case)




                          ...
Release the buildout


       What packages should be frozen ?

  - recipes

  - your released packages

  - exceptions (s...
Release the buildout

            authentication: use lovely.buildouthttp

[buildout]
...
extensions=lovely.buildouthttp
....
Release the buildout -> project layout

    project
       ...
       buildout
       packages
       release/0.1 <- tag f...
Release the buildout with collective.releaser :
with project_release

 $ cd buildout
 $ project_release
 What version you ...
Build your release distribution

  - bin/buildout on target system

  - remove some stuff

  - offline mode to ‘true’

  -...
Build your distribution with collective.releaser
with project_deploy

$ svn co http://somesvn/my_projet/releases/0.1 proje...
Upgrading the application




                79




                            79
Upgrading the application : 3 strategies


  #1 full release
  #2 diff release, using collective.releaser
  #3 egg pack re...
collective.releaser :

the releaser_project template
       project_release
        project_deploy
         project_eggs
 ...
That’s it !
 Thanks !
questions ?




               82




                    82
http://tarekziade.wordpress.com
zc.buildout @ PyPI
collective.dist @ PyPI
collective.releaser @ PyPI


http://alterway.fr
...
Upcoming SlideShare
Loading in …5
×

delivering applications with zc.buildout and a distributed model - Plone Conference 2008

26,794 views

Published on

Slides from Plone Conf 2008 (Washington D.C.)

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
26,794
On SlideShare
0
From Embeds
0
Number of Embeds
19,274
Actions
Shares
0
Downloads
89
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

delivering applications with zc.buildout and a distributed model - Plone Conference 2008

  1. 1. delivering applications with zc.buildout and a distributed model Tarek Ziadé <tarek.ziade@ingeniweb.com> 1
  2. 2. Who am I ? New to Plone (~1 year) -> I worked on peripheral matters Used to Zope (CPS core developer) CTO at Ingeniweb - a Plone company (>20 developers) Python fan - wrote some books about it (even in english, because you don’t hear my accent in a book) involved in plone.org migration, PSC, zc.buildout 2
  3. 3. It will look good on your desk besides Martin Aspeli’s book Special promotion during the PloneConf : buy one book == get a big hug 3
  4. 4. Thanks ! * Andreas Jung * Veda Williams * David Glick * Jeff Kowalczyk * Youenn Boussard * Christian Klinger * Jesse Snyder * Alec Mitchell * John Habermann * Maurits van Rees * Jean-François Roche * Martin Aspeli * Alain Meurant * Aleksandr Vladimirskiy * Jon Stahl * Alexander Limi * Stephen McMahon 4
  5. 5. Part 1 - working with packages Part 2 - working with zc.buildout Part 3 - application lifecycle 5
  6. 6. Part 1 - working with packages 6
  7. 7. distutils builds and distributes a package registers and uploads it to PyPI 7 7
  8. 8. get the screencast at http://ziade.org/ploneconf 8 8
  9. 9. +1 distutils is the standard in the whole Python community. 9 9
  10. 10. -1 distutils is broken (but there’s hope) 10 10
  11. 11. setuptools simple dependencies management, namespaced package egg distribution provides easy_install 11 11
  12. 12. “python setup.py bdist_egg” one single .egg file 12 12
  13. 13. quot;Eggs are to Pythons as Jars are to Java...quot; egg = deployment format in a zip archive 13 13
  14. 14. “sdist” is enough, though (and should be mandatory) 14 14
  15. 15. get the screencast at http://ziade.org/ploneconf 15 15
  16. 16. easy_install my_package = will get and install “my_package” from “PyPI” 16 16
  17. 17. +1 setuptools is the de facto standard in the Plone community. 17 17
  18. 18. -1 setuptools is broken too (pje says) (but there’s hope) 18 18
  19. 19. packaging future in Python ? 19 19
  20. 20. PJE Debian packagers BSD packagers Us ! Ian Zope 3 guys Repoze Discussions guys Python PEPs core TG2 guys developers Pylons guys Guido Heaven distutils 2 20 20
  21. 21. Packaging future in Python hard to know. but community-driven come sprint with us saturday ! 21 21
  22. 22. It’s OK, fellows we will eventually make it through. 22 22
  23. 23. Problems with packaging ? #1 PyPI == SPOF #2 packages need privacy sometimes #3 plone.org/products is *dying* 23 23
  24. 24. Solutions ? #1 PyPI == SPOF Make a PyPI mirror #2 packages need privacy sometimes Run you own private PyPI #3 plone.org/products is *dying* Make plone.org/products PyPI compatible 24 24
  25. 25. #1 Make a PyPI mirror : Smart mirroring easy_install collective.eggproxy 25 25
  26. 26. 26 26
  27. 27. In zc.buildout : the index option [buildout] index = http://my.mirror:8888 all calls will go through the proxy the mirror is filled on-demand 27 27
  28. 28. #2 Run your own private PyPI PloneSoftwareCenter ! 28 28
  29. 29. get the screencast on http://ziade.org/ploneconf 29 29
  30. 30. collective.dist python 2.6 new “register” and “upload” commands “mregister” and “mupload” in 2.4 and 2.5 $ python setup.py mregister sdist mupload $ python setup.py mregister sdist mupload -r plone.org $ python setup.py mregister sdist mupload -r my-company ... 30 30
  31. 31. Developer side : multiple servers with .pypirc [distutils] index-servers = pypi plone.org my-company [pypi] username:tarek password:hahaha [plone.org] username:tarek password:hahaha... repository:http://plone.org/products [my-company] username:tarek password:hahaha..hahaha.haha...ha..pfiiiuuuuuu..and..hahahaahah.i..am..dying.. repository:http://my-company.com/products 31 31
  32. 32. Let’s try our new PyPI with collective.dist 32 32
  33. 33. get the screencast on http://ziade.org/ploneconf 33 33
  34. 34. #3 Make plone.org/products PyPI compatible plone.org was suppose to switch to that for months. Lazy guys 34 34
  35. 35. Let’s try new.plone.org 35 35
  36. 36. get the screencast on http://ziade.org/ploneconf 36 36
  37. 37. The big picture 37 37
  38. 38. Yes !!! 38 developer #1 38
  39. 39. Yepeee !!! Releasing everywhere let’s spam PyPI and plone.org tchak ! tchak ! tchak ! developer #2 39 39
  40. 40. Hahaha ! We have our package all over the place. developer #3 40 40
  41. 41. end of part 1 questions ? #1 Make mirror #2 Run your own PyPI #3 push to several servers #3 use “mregister” and “mupload” 41 41
  42. 42. Part 2 - working with zc.buildout (reminders) 42 42
  43. 43. 5 hours in 2006 43 43
  44. 44. 5 hours in 2006 install python extra packages get zope install zope create an instance get extra products read extra products doc install extra products dependency install extra products doesn’t work ahhh right, install python-ldap checkout products in development doesn’t work ahhh right, wrong python-ldap version start to work 44 44
  45. 45. Me in 2006 45 45
  46. 46. 5 minutes in 2008 46 46
  47. 47. 5 minutes in 2008 get the “buildout” $ python bootstrap.py $ bin/buildout start to work 47 47
  48. 48. But that was not the main reason 48 48
  49. 49. 2005 Now zope.interface zope.schemas zope.stuff Zope one big-O zope.whatsthat package. “eggification” zope.makecoffee zope.dodishes ... over 100 packages 49 49
  50. 50. 2005 Now plone.ista plone.portlets plone.stuff Plone products plone.whatsthat “eggification” plone.makecoffee plone.enolp ... 50 50
  51. 51. Your app.site Killer Plone app.skin App ... 51 51
  52. 52. zope.stuff ... Your plone.stuff Distribution ... app.site app.skin 52 52
  53. 53. Python site-packages +100 packages to run Plone something is Increase conflict going wrong here probabilities 53
  54. 54. zc.buildout = isolated environment 54 54
  55. 55. zc.buildout best practices #1 use the same layout for all your projects #2 make sure all developers have the same environment #3 use one cfg per target 55 55
  56. 56. #1 same layout for all projects project1 docs buildout packages releases project2 docs buildout packages releases .... 56 56
  57. 57. easy_install collective.releaser 57 57
  58. 58. get the screencast on http://ziade.org/ploneconf 58 58
  59. 59. #2 make sure all developers have the same environment Warning Plone buildouts are source based Windows developers Get my Windows installer : python2.4.4-win32.zip Google “An installer for a buildout-ready Windows” 59 59
  60. 60. #3 use one cfg per target Typical buildout layout uses the extends feature buildout.cfg dev.cfg (extends buildout.cfg) prod.cfg (extends buildout.cfg) + bootstrap.py 60 60
  61. 61. buildout.cfg : [buildout] parts = one two [buildout] parts = dev.cfg : one two [buildout] three extends = buildout.cfg develop = parts = ... three develop = ... 61 61
  62. 62. demo creating a fresh Plone 3 buildout (Paste) adding the dev.cfg hooking a new development package adding a prod.cfg 62 62
  63. 63. get the screencast on http://ziade.org/ploneconf 63 63
  64. 64. end of part 2 questions ? #1 use the same layout for all projects #2 make sure all developers have the same environment #3 use one cfg per target 64 64
  65. 65. Part 3 - application lifecycle 65
  66. 66. 66
  67. 67. Releasing packages Releasing the buildout 67
  68. 68. Releasing packages for package in packages: raise the version edit CHANGES.txt create a branch (svn) go to that branch remove the dev tag (setup.cfg) release it with “mregister sdist mupload -r somehwere” release it with “mregister sdist mupload -r somehwereelse” 68 68
  69. 69. Releasing packages with collective.releaser: for package in packages: python setup.py release 69 69
  70. 70. Defining targets in .pypirc [distutils] index-servers = pypi plone.org private glob-style = true [pypi] ... release-command = mregister sdist mupload release-packages = iw.* [plone.org] ... release-command = mregister sdist mupload release-packages = plone.* collective.* [private] 70 ... 70
  71. 71. get the screencast on http://ziade.org/ploneconf 71 71
  72. 72. collective.releaser makes a lot of assumptions it’s svn-oriented (the first use case) 72 72
  73. 73. Release the buildout What packages should be frozen ? - recipes - your released packages - exceptions (security fixes, major bug fixes) 73 73
  74. 74. Release the buildout authentication: use lovely.buildouthttp [buildout] ... extensions=lovely.buildouthttp ... repository:http://my-company.com/products HOME/.buildout/.httpauth pypi,http://my-company.com/products,tarek,hahaha pypi,http://plone.org/products,tarek,hahaha 74 74
  75. 75. Release the buildout -> project layout project ... buildout packages release/0.1 <- tag for the buildout $ cd buildout $ svn cp . http://somewhere/releases/0.1 75 75
  76. 76. Release the buildout with collective.releaser : with project_release $ cd buildout $ project_release What version you are releasing ? 0.1 Added version file. 76 76
  77. 77. Build your release distribution - bin/buildout on target system - remove some stuff - offline mode to ‘true’ - tar -czvf release-0.1.tgz release/0.1 77 77
  78. 78. Build your distribution with collective.releaser with project_deploy $ svn co http://somesvn/my_projet/releases/0.1 project $ cd project $ project_deploy prod.cfg 78 78
  79. 79. Upgrading the application 79 79
  80. 80. Upgrading the application : 3 strategies #1 full release #2 diff release, using collective.releaser #3 egg pack release, using collective.releaser 80 80
  81. 81. collective.releaser : the releaser_project template project_release project_deploy project_eggs project_diff 81 81
  82. 82. That’s it ! Thanks ! questions ? 82 82
  83. 83. http://tarekziade.wordpress.com zc.buildout @ PyPI collective.dist @ PyPI collective.releaser @ PyPI http://alterway.fr 83 83

×