Big Fat FastPlone - Scale up, speed up

Text: Creative Commons Namensnennung-Keine
kommerzielle Nutzung- Keine Bearbeitung 3.0
Österreich Lizenz.1 von 40
12th Plone Conference
Bristol, UK, 2014
Big Fat Fast Plone
Scale Up, Speed Up
Jens W. Klein twitter: @yenzenz
jk@kleinundpartner.at
30.10.2014 http://kleinundpartner.at

I'am ...
● located in Innsbruck, Tyrol, Austria
● doing Plone since v.1.0
● a consultant, developer, open source evangelist
● owner of Klein & Partner KG, co-founder and
member of BlueDynamics Alliance
Ad:
Alpine City Sprint at Innsbruck
January 20th to 25th 2015 – registration at
http://www.coactivate.org/projects/alpinecitysprint2015

Problem
● Default Plone is not that fast
● Scales great horizontally (add machines)
● Still bottlenecks
– primary loading stuff from ZODB
– rendering of sites
– searching information
– third party services: ldap, crm, sap-bc, ...

You have
to know
your knobs
and
handles.
Juliett U-461

3 different customers
(1)
● Niederösterreich Kulturwirtschaft (NOEKU)
● central service for several cultural institutions in
Lower Austria
● over 30 Plone Sites (including e-ticketing)
● hi-availibility on low-to-mid budget
● self hosted on hardware,
● VM's with KVM, HA with Pacemaker, CRM, DRBD

Noeku in Detail

(2)
● Zumtobel Group AG
● Internationally leading supplier of integral
lighting solutions
● Brand specific international product Portals
● Customer Extranets, B2B e-shop
● Hosting on dedicated and managed VM's and
NAS storage

(3)
● HTU Graz
● general students' representative body, serves
as the students' government, at the University
of Graz
● One plone site with several subsites (lineage)
● Lots of students looking at it = peak-load
● One virtual machine on overloaded host
= not enough resources
● Low-budget

Data Flow in Plone
or
the Publishing Chain
Database
Client
Connection
Pool
Plone 1..n
Caching
Proxy
Load
Balancer
Webserver
Webbrowser

Data Flow in Plone
or
Database
Client
Connection
Pool
Plone 1..n
Caching
Proxy
Load
Balancer
Webserver
Webbrowser
ZEO-Server or
PostgreSQL, plus
Blobstorage (NFS, NAS)
Different ZEO
Storage Adapters
HAProxy or
Pound
Varnish
(what else?)
NGINX

Data Flow in Plone
or
Database
Client
Connection
Pool
Plone 1..n
Caching
Proxy
Load
Balancer
Webserver
Webbrowser
ZEO-Server or
PostgreSQL, plus
Different ZEO
Storage Adapters
HAProxy or
Pound
Varnish
(what else?)
NGINX
Memcached
LDAP
Other third part services

Data Flow in Plone
or
Database
Client
Connection
Pool
Plone 1..n
Caching
Proxy
Load
Balancer
Webserver
Webbrowser
ZEO-Server or
PostgreSQL, plus
Different ZEO
Storage Adapters
HAProxy or
Pound
Varnish
(what else?)
NGINX
Memcached
LDAP
Other third part services
Consider also:
CPU Cores
Used RAM/ Diskcache ratio
IO network and disk
...

Tools
know them!
● install Munin on every node and monitor all you
can.
● fio is simple powerful to get measures
● read how linux manages disk/RAM
● know your hardware and your VMs (if any)

Database Level
● NOEKU: ZEO-Server + BlobStorage, both
replicated with DRBD on own self-managed VM
● Zumtobel: RelStorage on PostgreSQL and
Blobs from NAS over NFS
● HTU Graz: RelStorage on PostgreSQL, all on
one machine.

Never store blobs in ZODB
Use blobstorage!
(check your code and adons!)
First things first

Classical ZEO
Server+Blobstorage
● Using classical ZEO-server + blobstorage
– ensure a fast IO to HDD or NAS
– have enough RAM for disk buffering
● Blobstorage on NFS/NAS:
– shared blobs and mount on each node
– mount NAS/NFS read-only on webserver node
and use collective.xsendfile (X-HTTP-Accel)

RelStorage + Blobstorage
● Never store blobs in SQL DB (same as with
ZODB, see slide before)
● Do not use MySQL if you can avoid it
● Configure your SQL DB properly, defaults are
not enough!
● Ensure a sane disk-/storage-IO (on Vms this is a
common bottleneck).
● Have RAM left for OS-level disk-caching.

Connections Pool
ZEO- vs. RelStorage Adapter
● ZEO-Server pushes
invalidations to client
● Disk cache of pickled
objects per Zope
instance
● RAM-cache of
unpickled objects per
connection
● ZEO-Client polls for
invalidated objects
● Memcached of pickled
objects shared
between Zope
instances on one
machine
● RAM-cache of
unpickled objects per
connection

Database-/ Client-
Configurations (1)
● NOEKU (ZEO)
– 30 sites with different sizes
– 2 (rare: 4) instances, each 2 threads
– RAM cache ~750 - 20000 objects – depending on
the site – no disk cache
– ZEO-Server with DRBD and failover (works with
Plone >=4) on own Virtual Maschine

Database-/ Client-
Configurations (2)
● Zumtobel (RelStorage, history free)
– Different setups for different kinds of sites!
– Always 2 VMs for, because of easier
maintainability
– 16 instances plus some worker-instances (async
import, zipfile creation, ...)
– each 2 or 4 threads
– RAM cache 30000 or 100000 objects
– Memcached as shared connection cache
– relstorage_packer (search on pypi)

Database-/ Client-
Configurations (3)
● HTU (RelStorage, history free)
– 6 instances, each 1 thread
– RAM cache 30000 objects
– Memcached as shared connection cache
– Poll interval 120 sec
– Blobstorage shared folder on same maschine

All between Request and
Response has to be fast
REQUEST
(what Plone does)
RESPONSE

Plone Configuration
● Turn off: debug mode, verbose security, logging,
deprecation warnings, …
● Configure plone.app.caching, even if you do not
plan to use varnish! → browser caches too!
● multiple plone instances: use memcached
instead of classic ramcache
● install jarn.checkinterval and tweak the
checkinterval of your instance in your live
buildout.

Custom Code and
Someones Add-On
● know plone.memoize, use it
● never calculate/search twice
● cache searched/ prepared data and/or rendered
pages or snippets
● again: never store blobs in ZODB.
● use the catalog
● dont overuse metadata

Write conflicts
● Happens >90% of time at catalog
● Use collective.solr/ collective.indexing
● Reduce time of the transaction

Lots of objects?
● large site with several 100 thousands of content
items (and even more objects in ZODB)?
● Catalog queries are slow?
● Use an own mount point with own caches for
the catalog
● if that does not help, use collective.solr

Archetypes vs. Dexterity
● in Archetypes avoid to "wake up" (read: load
from DB) the object, ask the catalog and use
metadata
● in Dexterity its some times cheaper to wake up
the object:
– If objects are small.
– And if you iterate over a folder or subtree
– Or if adding lots of metadata to the catalog
would be needed.

Using Webservices, LDAP,
other Databases
● Talking to external systems over network is
slow.
● Cache whats possible → plone.memoize
● Measure the external system!
● How fail safe are the services?
● Does your addon/custom code behaves nicely if
the service fails?

In case of serious
performance trouble
● Measure!
– Munin (base system parameters),
– Fio,
– collective.traceview
– Good old friend Products.ZopeProfiler
– haufe.requestmonitoring,
Products.LongRequestLogger
● Do not change more than one parameter at
once,
● measure again.

plone.app.caching
● Caching is essential in order to serve hi-traffic.
● On Peak load a site can be faster (!) than on
normal traffic.
● Varnish + plone.app.caching are our superheros
● Default settings are good for default sites
● Custom Addons with own types and
templates needs extra configuration for
each type and template:
– Do this! Calculate time for this task, its some
work!

Micro-Caching
● On hi traffic introduce a new caching rule for
one two five minute caches
● It speeds up a public facing site really
good!
● Apply this rule to all types/ templates youre
expecting to be hit
● This means a small delay for content editors
between save and live.

Microcaching
used at...
● Zumtobel
– Extranet on group level
– Portal for all anonymous
● HTU Graz
– For all anonymous
● Bunch of other sites
– Mostly anonymous access

Load Balancer
● Pound → tiny, simple, stable, old, but difficult to
get measurements out of it.
● HAProxy → not that simple, complex
configuration, newer, nice WebUI for stats
● Point same request type to same instance
– Stick to session (public site)
– Stick to User (intranet)
– Stick to Group/ other vector (customer extranet)
● Group Instances for editors/ anonymous

Webserver
● NGINX – others possible, but more difficult to
configure
● Set the proxy_* to recommended values
● Consider configuring direct blob delivery (using
collective.xsendfile)
– collective.xsendfile needs some love to work fine
with dexterity and plone.app.imaging (deliver
more variants) – any takers?
● Mostly it works fine OOTB

*Cat Image (Bobcat by Markus Wollny)
CC BY-NC-SA2.0 License

Further Reading
● Plone Documentation Chapter
„Performance and tuning“
http://docs.plone.org/manage/deploying/testing
_tuning/performance/index.html

Questions?

Big Fat FastPlone - Scale up, speed up

More Related Content

What's hot

Viewers also liked

Similar to Big Fat FastPlone - Scale up, speed up

More from Jens Klein

Recently uploaded

Big Fat FastPlone - Scale up, speed up