Deployment & Continuous integration
of a Plone/Zope application
Julien Pivotto

PyconFR’13
October 26, 2013

;
whoami
sysadmin @ inuits
• open-source defender for 7+ years
• devops believer
• @roidelapluie on twitter/github
•

Julien...
•
•
•

Plone is an Open-Source CMS
Written in python
Vibrant community

Julien Pivotto

Deployment and CI of a Plone/Zope ...
•
•
•

Open-Source Web application server
Written in Python
Used by Plone

Julien Pivotto

Deployment and CI of a Plone/Zo...
Plone/Zope application
•
•
•

Threads = instances
Built using Buildout
Modules (eggs)

Julien Pivotto

Deployment and CI o...
The mission
•
•
•
•

Several Zope/Plone applications
Scalability
Automation
Reliability

Julien Pivotto

Deployment and CI...
The old days
•
•
•
•

Manual work
Failover HA
Supervisord
No overview

Julien Pivotto

Deployment and CI of a Plone/Zope a...
BUT
Run buildout on each server/environment
• Compilation error in the middle
• Pypi is down
• Which commit is faulty?
•

...
The goal
•
•
•
•
•

Less failure during builds
More testing
Puppetize all the things
Reproduce all the builds
More env: de...
Infrastructure changes
• DRBD for PostgreSQL ⇒ PostgreSQL 9 native replication
• DRBD for files ⇒ GlusterFS for data, RPM f...
Puppetization
•
•
•
•

From nothing to everything
Plone, but also apache, haproxy
. . . and anything else
Mcollective to o...
Development decisions
•
•
•

Move away from SVN
Write tests
Automate them in Jenkins

Julien Pivotto

Deployment and CI of...
It takes time
•
•
•
•

Cultural shift
Legacy stuff
A lot of surprises
Compromises between Dev and Ops

Julien Pivotto

Depl...
Jenkins

•
•
•
•

An open-source CI server
More than 600 plugins
Flexible
Jobs & pipelines
Julien Pivotto

Deployment and ...
Julien Pivotto

Deployment and CI of a Plone/Zope application

;
Jobs
•
•
•
•

Reproducible builds
Single place to find build logs
History of builds
Creating (RPM) artifacts

Julien Pivott...
Pipeline
•
•
•

Build is only part of the process
Need to package, deploy, . . .
Running tests

Julien Pivotto

Deployment...
Tests
•
•
•
•
•
•

Running tests on each module
Feedback: graphes, mails
Code coverage
Violations
Unit tests
zc.recipe.tes...
Translation in Jobs
virtualenv --no-site-packages .
bin/python bootstrap.py
bin/buildout -c jenkins.cfg buildout:eggs-dire...
Julien Pivotto

Deployment and CI of a Plone/Zope application

;
Buildout
•
•
•
•

Running buildout in a Spec file
Creating a RPM
Using virtualenv
Starting from scratch each time

Julien P...
Ops like RPM
•
•
•
•
•
•

Easy to deploy anywhere
Consistency and dependencies
Find where a file comes from
Orchestrate wit...
Time

Julien Pivotto

Deployment and CI of a Plone/Zope application

;
Deploying Zope
•
•
•
•
•

Building RPM
Adding them to a yum repo (with pulp)
Cleaning yum cache (with mcollective)
Running...
Reports
•
•
•
•

Jenkins build log
Puppet logs
Zope instances logs
IRC, mails, . . .

Julien Pivotto

Deployment and CI of...
Role of Puppet
•
•
•
•

Puppet creates the configuration
Update the package
Restarts the instances
Puppet (hiera) creates p...
Database upgrades
•
•
•

WIP, no solution chosen yet
Postscript in RPM?
Commands trigerred by puppet?

Julien Pivotto

Dep...
Logs
Zope/Plone: logs files/instances
• 10 instances * 2 servers
⇒ 20 log files
• Plus apache logs
• Plus system logs
•

Jul...
The deprecated way
•
•
•

tail -f
multitail
clusterssh

Julien Pivotto

Deployment and CI of a Plone/Zope application

;
Logstash
•
•
•
•
•

An open source tool
Manage your logs
Input from anywhere
Filter
Output to anywhere
Julien Pivotto

Dep...
Logstash

Julien Pivotto

Deployment and CI of a Plone/Zope application

;
Plone and Logstash
•
•
•
•

Plone output to syslog
Mangle to add good fields
Logstash output to ElasticSearch
Kibana3 creat...
zope.conf
<syslog>
address 172.16.28.55:5544
facility local3
format %(asctime)s ZopApp-Server zope[%(process)s]:
instance1...
filter {
grok {
type
=> "syslog"
pattern => ["(?m)<%= "<%" %>{POSINT:syslog_pri}>
%{SYSLOGTIMESTAMP:syslog_timestamp}
%{SY...
frametitle{zope.conf}
filter {
grok {
tags
=> ["syslog-zope"]
pattern => ["(?m)(?<instance>[a-z0-9]+)
[(?<loglevel>[A-Z]+)...
Kibana3
•
•
•
•
•

Plays with ElasticSearch and Logstash
Easily build dashboards
Share the metric and logs with anyone
Fil...
Julien Pivotto

Deployment and CI of a Plone/Zope application

;
Conclusions
This is a sysadmin point of view
• Still a lot of work to be done
• From a "black box"
. . . To a fully automa...
This work is being realized with
Affinitic SPRL, Plone developers in Belgium
http://affinitic.be

Julien Pivotto

Deployment a...
Thank you
Any question?

Julien Pivotto

Deployment and CI of a Plone/Zope application

;
Contact
Julien Pivotto
julien@inuits.eu
@roidelapluie

INUITS bvba
INUITS bvba
Belgium
Belgium
+32 473 441 636
+32 473 441...
Upcoming SlideShare
Loading in...5
×

Deployment and Continous Integration of a Zope/Plone application

1,758

Published on

Conference given at PyconFR'13

Published in: Technology
2 Comments
4 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,758
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
26
Comments
2
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "Deployment and Continous Integration of a Zope/Plone application"

  1. 1. Deployment & Continuous integration of a Plone/Zope application Julien Pivotto PyconFR’13 October 26, 2013 ;
  2. 2. whoami sysadmin @ inuits • open-source defender for 7+ years • devops believer • @roidelapluie on twitter/github • Julien Pivotto Deployment and CI of a Plone/Zope application ;
  3. 3. • • • Plone is an Open-Source CMS Written in python Vibrant community Julien Pivotto Deployment and CI of a Plone/Zope application ;
  4. 4. • • • Open-Source Web application server Written in Python Used by Plone Julien Pivotto Deployment and CI of a Plone/Zope application ;
  5. 5. Plone/Zope application • • • Threads = instances Built using Buildout Modules (eggs) Julien Pivotto Deployment and CI of a Plone/Zope application ;
  6. 6. The mission • • • • Several Zope/Plone applications Scalability Automation Reliability Julien Pivotto Deployment and CI of a Plone/Zope application ;
  7. 7. The old days • • • • Manual work Failover HA Supervisord No overview Julien Pivotto Deployment and CI of a Plone/Zope application ;
  8. 8. BUT Run buildout on each server/environment • Compilation error in the middle • Pypi is down • Which commit is faulty? • Julien Pivotto Deployment and CI of a Plone/Zope application ;
  9. 9. The goal • • • • • Less failure during builds More testing Puppetize all the things Reproduce all the builds More env: dev, test, demo, prod Julien Pivotto Deployment and CI of a Plone/Zope application ;
  10. 10. Infrastructure changes • DRBD for PostgreSQL ⇒ PostgreSQL 9 native replication • DRBD for files ⇒ GlusterFS for data, RPM for code • Supervisord to launch instances ⇒ Puppet to start services • Pound as reverse proxy ⇒ Haproxy • Internal mirror of pypi Julien Pivotto Deployment and CI of a Plone/Zope application ;
  11. 11. Puppetization • • • • From nothing to everything Plone, but also apache, haproxy . . . and anything else Mcollective to orchestrate Julien Pivotto Deployment and CI of a Plone/Zope application ;
  12. 12. Development decisions • • • Move away from SVN Write tests Automate them in Jenkins Julien Pivotto Deployment and CI of a Plone/Zope application ;
  13. 13. It takes time • • • • Cultural shift Legacy stuff A lot of surprises Compromises between Dev and Ops Julien Pivotto Deployment and CI of a Plone/Zope application ;
  14. 14. Jenkins • • • • An open-source CI server More than 600 plugins Flexible Jobs & pipelines Julien Pivotto Deployment and CI of a Plone/Zope application ;
  15. 15. Julien Pivotto Deployment and CI of a Plone/Zope application ;
  16. 16. Jobs • • • • Reproducible builds Single place to find build logs History of builds Creating (RPM) artifacts Julien Pivotto Deployment and CI of a Plone/Zope application ;
  17. 17. Pipeline • • • Build is only part of the process Need to package, deploy, . . . Running tests Julien Pivotto Deployment and CI of a Plone/Zope application ;
  18. 18. Tests • • • • • • Running tests on each module Feedback: graphes, mails Code coverage Violations Unit tests zc.recipe.testrunner Julien Pivotto Deployment and CI of a Plone/Zope application ;
  19. 19. Translation in Jobs virtualenv --no-site-packages . bin/python bootstrap.py bin/buildout -c jenkins.cfg buildout:eggs-directory= /var/lib/jenkins/buildout-shared-eggs bin/test-all Julien Pivotto Deployment and CI of a Plone/Zope application ;
  20. 20. Julien Pivotto Deployment and CI of a Plone/Zope application ;
  21. 21. Buildout • • • • Running buildout in a Spec file Creating a RPM Using virtualenv Starting from scratch each time Julien Pivotto Deployment and CI of a Plone/Zope application ;
  22. 22. Ops like RPM • • • • • • Easy to deploy anywhere Consistency and dependencies Find where a file comes from Orchestrate with puppet/mcollective Version number Conf does not belong to package Julien Pivotto Deployment and CI of a Plone/Zope application ;
  23. 23. Time Julien Pivotto Deployment and CI of a Plone/Zope application ;
  24. 24. Deploying Zope • • • • • Building RPM Adding them to a yum repo (with pulp) Cleaning yum cache (with mcollective) Running puppet (with mcollective) dev -> staging -> demo and prod Julien Pivotto Deployment and CI of a Plone/Zope application ;
  25. 25. Reports • • • • Jenkins build log Puppet logs Zope instances logs IRC, mails, . . . Julien Pivotto Deployment and CI of a Plone/Zope application ;
  26. 26. Role of Puppet • • • • Puppet creates the configuration Update the package Restarts the instances Puppet (hiera) creates password files Julien Pivotto Deployment and CI of a Plone/Zope application ;
  27. 27. Database upgrades • • • WIP, no solution chosen yet Postscript in RPM? Commands trigerred by puppet? Julien Pivotto Deployment and CI of a Plone/Zope application ;
  28. 28. Logs Zope/Plone: logs files/instances • 10 instances * 2 servers ⇒ 20 log files • Plus apache logs • Plus system logs • Julien Pivotto Deployment and CI of a Plone/Zope application ;
  29. 29. The deprecated way • • • tail -f multitail clusterssh Julien Pivotto Deployment and CI of a Plone/Zope application ;
  30. 30. Logstash • • • • • An open source tool Manage your logs Input from anywhere Filter Output to anywhere Julien Pivotto Deployment and CI of a Plone/Zope application ;
  31. 31. Logstash Julien Pivotto Deployment and CI of a Plone/Zope application ;
  32. 32. Plone and Logstash • • • • Plone output to syslog Mangle to add good fields Logstash output to ElasticSearch Kibana3 creates dashboards Julien Pivotto Deployment and CI of a Plone/Zope application ;
  33. 33. zope.conf <syslog> address 172.16.28.55:5544 facility local3 format %(asctime)s ZopApp-Server zope[%(process)s]: instance1 [%(levelname)s] %(name)s | %(message)s dateformat %b %d %H:%M:%S level info </syslog> Julien Pivotto Deployment and CI of a Plone/Zope application ;
  34. 34. filter { grok { type => "syslog" pattern => ["(?m)<%= "<%" %>{POSINT:syslog_pri}> %{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program} (?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}"] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{@source_host}" ] add_tag => "syslog-%{syslog_program}" } } Julien Pivotto Deployment and CI of a Plone/Zope application ;
  35. 35. frametitle{zope.conf} filter { grok { tags => ["syslog-zope"] pattern => ["(?m)(?<instance>[a-z0-9]+) [(?<loglevel>[A-Z]+)] (?<loggername>[A-Za-z0-9.]+) | %{GREEDYDATA:logmessage}"] add_tag => "grokked-zope" } mutate { tags => ["grokked-zope"] rename => [ "@source_host", "application" ] replace => [ "@message", "%{logmessage}" ] add_tag => "loggername-%{loggername}" } } Julien Pivotto Deployment and CI of a Plone/Zope application ;
  36. 36. Kibana3 • • • • • Plays with ElasticSearch and Logstash Easily build dashboards Share the metric and logs with anyone Filter by time, instance, http code. . . View all the logs at the same place Julien Pivotto Deployment and CI of a Plone/Zope application ;
  37. 37. Julien Pivotto Deployment and CI of a Plone/Zope application ;
  38. 38. Conclusions This is a sysadmin point of view • Still a lot of work to be done • From a "black box" . . . To a fully automated environment • Not a unique solution • You need to get it working for you . . . in your way • Julien Pivotto Deployment and CI of a Plone/Zope application ;
  39. 39. This work is being realized with Affinitic SPRL, Plone developers in Belgium http://affinitic.be Julien Pivotto Deployment and CI of a Plone/Zope application ;
  40. 40. Thank you Any question? Julien Pivotto Deployment and CI of a Plone/Zope application ;
  41. 41. Contact Julien Pivotto julien@inuits.eu @roidelapluie INUITS bvba INUITS bvba Belgium Belgium +32 473 441 636 +32 473 441 636 https://inuits.eu https://inuits.eu Julien Pivotto Deployment and CI of a Plone/Zope application ;
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×