1. Deployment & Continuous integration
of a Plone/Zope application
Julien Pivotto
PyconFR’13
October 26, 2013
;

2. whoami
sysadmin @ inuits
• open-source defender for 7+ years
• devops believer
• @roidelapluie on twitter/github
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

3. •
•
•
Plone is an Open-Source CMS
Written in python
Vibrant community
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

4. •
•
•
Open-Source Web application server
Written in Python
Used by Plone
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

5. Plone/Zope application
•
•
•
Threads = instances
Built using Buildout
Modules (eggs)
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

6. The mission
•
•
•
•
Several Zope/Plone applications
Scalability
Automation
Reliability
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

7. The old days
•
•
•
•
Manual work
Failover HA
Supervisord
No overview
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

8. BUT
Run buildout on each server/environment
• Compilation error in the middle
• Pypi is down
• Which commit is faulty?
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

9. The goal
•
•
•
•
•
Less failure during builds
More testing
Puppetize all the things
Reproduce all the builds
More env: dev, test, demo, prod
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

10. Infrastructure changes
• DRBD for PostgreSQL ⇒ PostgreSQL 9 native replication
• DRBD for files ⇒ GlusterFS for data, RPM for code
• Supervisord to launch instances ⇒ Puppet to start services
• Pound as reverse proxy ⇒ Haproxy
• Internal mirror of pypi
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

11. Puppetization
•
•
•
•
From nothing to everything
Plone, but also apache, haproxy
. . . and anything else
Mcollective to orchestrate
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

12. Development decisions
•
•
•
Move away from SVN
Write tests
Automate them in Jenkins
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

13. It takes time
•
•
•
•
Cultural shift
Legacy stuff
A lot of surprises
Compromises between Dev and Ops
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

14. Jenkins
•
•
•
•
An open-source CI server
More than 600 plugins
Flexible
Jobs & pipelines
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

15. Julien Pivotto
Deployment and CI of a Plone/Zope application
;

16. Jobs
•
•
•
•
Reproducible builds
Single place to find build logs
History of builds
Creating (RPM) artifacts
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

17. Pipeline
•
•
•
Build is only part of the process
Need to package, deploy, . . .
Running tests
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

18. Tests
•
•
•
•
•
•
Running tests on each module
Feedback: graphes, mails
Code coverage
Violations
Unit tests
zc.recipe.testrunner
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

19. Translation in Jobs
virtualenv --no-site-packages .
bin/python bootstrap.py
bin/buildout -c jenkins.cfg buildout:eggs-directory=
/var/lib/jenkins/buildout-shared-eggs
bin/test-all
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

20. Julien Pivotto
Deployment and CI of a Plone/Zope application
;

21. Buildout
•
•
•
•
Running buildout in a Spec file
Creating a RPM
Using virtualenv
Starting from scratch each time
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

22. Ops like RPM
•
•
•
•
•
•
Easy to deploy anywhere
Consistency and dependencies
Find where a file comes from
Orchestrate with puppet/mcollective
Version number
Conf does not belong to package
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

23. Time
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

24. Deploying Zope
•
•
•
•
•
Building RPM
Adding them to a yum repo (with pulp)
Cleaning yum cache (with mcollective)
Running puppet (with mcollective)
dev -> staging -> demo and prod
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

25. Reports
•
•
•
•
Jenkins build log
Puppet logs
Zope instances logs
IRC, mails, . . .
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

26. Role of Puppet
•
•
•
•
Puppet creates the configuration
Update the package
Restarts the instances
Puppet (hiera) creates password files
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

27. Database upgrades
•
•
•
WIP, no solution chosen yet
Postscript in RPM?
Commands trigerred by puppet?
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

28. Logs
Zope/Plone: logs files/instances
• 10 instances * 2 servers
⇒ 20 log files
• Plus apache logs
• Plus system logs
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

29. The deprecated way
•
•
•
tail -f
multitail
clusterssh
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

30. Logstash
•
•
•
•
•
An open source tool
Manage your logs
Input from anywhere
Filter
Output to anywhere
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

31. Logstash
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

32. Plone and Logstash
•
•
•
•
Plone output to syslog
Mangle to add good fields
Logstash output to ElasticSearch
Kibana3 creates dashboards
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

33. zope.conf
<syslog>
address 172.16.28.55:5544
facility local3
format %(asctime)s ZopApp-Server zope[%(process)s]:
instance1 [%(levelname)s] %(name)s | %(message)s
dateformat %b %d %H:%M:%S
level info
</syslog>
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

34. filter {
grok {
type
=> "syslog"
pattern => ["(?m)<%= "<%" %>{POSINT:syslog_pri}>
%{SYSLOGTIMESTAMP:syslog_timestamp}
%{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}
(?:[%{POSINT:syslog_pid}])?:
%{GREEDYDATA:syslog_message}"]
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{@source_host}" ]
add_tag => "syslog-%{syslog_program}"
}
}
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

35. frametitle{zope.conf}
filter {
grok {
tags
=> ["syslog-zope"]
pattern => ["(?m)(?<instance>[a-z0-9]+)
[(?<loglevel>[A-Z]+)]
(?<loggername>[A-Za-z0-9.]+) |
%{GREEDYDATA:logmessage}"]
add_tag => "grokked-zope"
}
mutate {
tags
=> ["grokked-zope"]
rename => [ "@source_host", "application" ]
replace => [ "@message", "%{logmessage}" ]
add_tag => "loggername-%{loggername}"
}
}
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

36. Kibana3
•
•
•
•
•
Plays with ElasticSearch and Logstash
Easily build dashboards
Share the metric and logs with anyone
Filter by time, instance, http code. . .
View all the logs at the same place
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

37. Julien Pivotto
Deployment and CI of a Plone/Zope application
;

38. Conclusions
This is a sysadmin point of view
• Still a lot of work to be done
• From a "black box"
. . . To a fully automated environment
• Not a unique solution
• You need to get it working for you
. . . in your way
•
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

39. This work is being realized with
Affinitic SPRL, Plone developers in Belgium
http://affinitic.be
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

40. Thank you
Any question?
Julien Pivotto
Deployment and CI of a Plone/Zope application
;

41. Contact
Julien Pivotto
julien@inuits.eu
@roidelapluie
INUITS bvba
INUITS bvba
Belgium
Belgium
+32 473 441 636
+32 473 441 636
https://inuits.eu
https://inuits.eu
Julien Pivotto
Deployment and CI of a Plone/Zope application
;