Modern, scalable deployment for plone

Scalable, modern
deployment for Plone
Christian Theune
ct@gocept.com

Tuesday, October 16, 12

Infrastructure


Entropy

Platform
Application
Components
Management
Automation

Monitoring

Support OS
Virtualization
Hardware
Data center

Data center


Hardware


Virtualization OS

Components

• Virtualization is very
affordable

• Platform creates
interface between app
and infrastructure


Automation

• convergence

• idempotency

• versioning

• everything as code

• continuous repairs


Management

• Take care of updates and “production-
ready” conﬁguration
• Backup and other day-to-day operations
• OS and regularly used components
• Gradual process moving components from
user-space installations to platform-wide
features


Monitoring

• Nagios :(

• (Un-)fortunately:
#monitoringsucks


Support

• Request tracker

• custom glue code

• Daily review

• Interrupt shield


Application deployment
and operations


Deployment

• Quick
• Repeatable
• Platform-independent
• single-click


Batou
• Model-based

• Use everywhere

• No effect on runtime

• Orchestration

• Convergency

• Idempotency

• Reusability


class Solr(Component):

def configure(self):
self.address = Address(self.host.fqdn, 9000)
self.provide('solr:server', self.address)

self += Buildout('solr', python='2.7')

self += Program('solr',
command='java',
command_absolute=False,
args='-jar start.jar',
directory=self.expand(
'{{component.workdir}}/parts/instance'),
restart=True)


[hosts]
oshaweb00 = nginx, www, varnish,
supervisor, haproxy, docs
oshaweb01 = zope, hwthemes, supervisor
...
oshaweb20 = zope, hwthemes, supervisor
oshaweb21 = zeo, openldap, supervisor
oshaweb22 = nginx, media2
oshaweb23 = nginx, birt, lms, ugm, supervisor
oshaweb24 = solr, memcached, supervisor
oshaweb26 = mailout

[component:haproxy]
strategy = production_strategy
bots = 3
whitelist = 8
default = 3
...

Components
good bad
nginx apache
varnish mysql, BDB
haproxy OpenLDAP
PostgreSQL
memcached
supervisor
postﬁx


Minimal "production-
ready" checklist
• Sane front-end conﬁguration
• System startup/shutdown
• Log rotation
• Database maintenance
• Monitoring: processes and ports
• Service user

Secrets
• One AES-encrypted config file per
environment
• Transparent decryption
• Safety-belt to avoid accidental checking in
unencrypted files
• Future: PGP-encryption


Environments

• Production vs. Staging
• Structurally identical but smaller
• Only one of many (dev, integration, ...)
• "Fly what you test, test what you ﬂy."


Web applications

• Shared nothing in your app!
• Many small processes
• Do not listen early
• Never store runtime conﬁg in the database


Web applications II

• Log long running requests
• Avoid synchronous external requests
• Avoid accidental write requests.
• Use feature switches.


Process control
• only one component to
integrate

• turn anything into a
daemon

• use superlance (httpok,
memmon) for active
restarts

• use 3.0 release candidate


Python: virtualenv and
buildout

• use both, combined
• buildout might be a hammer, but not
everything is a nail


more buildout
• use extends to keep your conﬁgs clean and
structured
• allow-picked-versions=False
• -t 3
• buildout >= 1.6
• mr.developer
• allowed-hosts = ...


and even more buildout
• multiple small better than one big
• different Python versions
• different subsets of egg versions
• use mirrors - see pypi-mirrors.org
• latency to the PyPI server is the biggest
factor in buildout performance


Databases

• As much RAM as you can afford - seriously
• Low latency network (<0.1ms)
• Multi-core if necessary
• Disk usage < 40%


story: more hardware =
slower
• Application is slow.
• Customer buys more hardware.
Speciﬁcally: RAM.
• System is slower now.
• System runs many components. Hard to
see through.


slower
• One number stands out: process exit rate
is 20/sec. PosgreSQL. What?
• Customer creates new connection for
each query.

• PostgreSQL forks master for new
connection.
• Fork went from few 10ms to few 100ms

slower

• Diagnose. Find root cause.


HTTP/S termination
• HTTP and SSL

• Virtual hosting

• Redirects

• host normalization

• Proxy to load balancer

• Static ﬁles


Load balancing
• LB + HA

• Classify requests

• redispatch + retries

• balance: leastconn

• minconn 1, maxconn 2, threads +2

• Live statistic UI


Monitoring

• again :(

• processes, ports, logs

• model dependencies

• check_webpage.rb

• Business processes (BPI)


Reporting
• report + phone call

• incidents

• changes

• availability

• performance

• trends


SLAs
• Availability
• more keeps your business safe
• less reduces costs
• Friends don't let friends think in "nines".
• Alignment of technical and organisational
measures


References

• "The practice of system and network
administration" (Limoncelli, et. al.)
• Mark Burgess on conﬁg management
• "The twelve-factor app" (Heroku)
• gocept.net/doc


Conclusion I

• Use infrastructure to build platforms
• Platforms should be transparent.
• Deploy applications on platforms.


Conclusion II

• Do not cobble together random stuff from
the internet.
• Careful work pays. Buzzwords don't.
• If you hear "enterprise". Run.
• Sapere aude.


Conclusion III

• I don't think IaaS and PaaS in the current
extremes are a good answer to many
projects.
• Many people experiment with ideas.
Everything is becoming more ﬂexible.
• Let's play!


Questions?

Modern, scalable deployment for plone

More Related Content

Similar to Modern, scalable deployment for plone

Recently uploaded

Modern, scalable deployment for plone