ExpressionEngine Performance &
Optimization:
Laying a Solid Foundation (& Caching)
CHR IS WE L L S – CE O – NE XCE SS .NE T L LC
The Ambassador Bridge (USA/Canada)
• Completed in 1929 (Longest suspension bridge worldwide)
• Busiest international trade crossing in North America!
• Carries 25% of all US/Canadian merchandise trade

• Drive south to go north?
• Owned by a private citizen !?
The Ambassador Bridge (USA/Canada)

The DetroitWindsor tunnel
Detroit, MI USA
is cool too!

Windsor, Ontario
Canada
Today’s Topics
• Why is the foundation important?
• MySQL choices

• PHP & the Apache/PHP hand-off
• Caching overview
• ExpressionEngine internal caching

• ExpressionEngine (modified) template caching
• Caching with CE cache
• Caching the caches with Varnish

• Clustering ExpressionEngine
Perspective
• Nexcess is a 13 year old company
• 90 employees (customer service being the largest contingent)
• Staff of 48 in support/customer service (all U.S./Michigan based)

• Wholly owned data centers in the U.S.
• Partner data centers in the U.K. and Australia
• For better or worse we are not designers
• Hosting company / infrastructure view of the system
• We often are developers but again with a view that differs from web
application developers
Why Care About The Foundation?
• Security, Stability, Performance
• Proper security/stability/performance permeates to the bare metal

• Do what you can with the access you have, ask questions about what is out of reach
• Assume nothing
The Foundation
• Linux
• Reboot-less updates (including on the kernel level)

• Apache (of Nginx)
• MySQL (Percona Server)
• PHP
• ExpressionEngine
MySQL (For Very High Traffic Sites)
• Don’t use MySQL!
• Use Percona Server!

• Percona Server is a simple MySQL replacement
• Percona is much better under stress
• Percona has a very good implementation of InnoDB tables (called XtraDB)

• Use InnoDB!
• Row (instead of table) level locking
• Tune InnoDB for your CPU / RAM configuration – it matters!
• A sample my.cnf is on our github account: https://github.com/nexcess/ee-whitepaperv1-configs

• MariaDB is an alternative as well although we haven’t benchmarked it (yet)
PHP Basics
• Use APC (as an opcode cache only)
• apc.shm_size
• apc.num_files_hint
• apc_stat

= 256M (at least)
= 10000 (at least)
= 0 (for production)

• Bump PHP’s memory_limit (128M works well)
• Turn off open_basedir
• Leaving it on kills the realpath cache
• Security concerns can be handled with tighter file system security

• Use a recent version of PHP (5.4.x+)
• 5.4.x is fast!
• 5.4.x has a much lower memory footprint than prior versions

• Invoke PHP in the right way
The PHP Hand-off
• This handoff is critical for both security and performance
• Apache + mod_php
• Runs as webserver user
• Unified mega-process

• Apache + suPHP / phpSuEXEC
• Runs as you
• Expensive to create

• Apache + PHP-FPM
• Runs as you

• Cheap to use
• Scales more efficiently than all of the above
The PHP Hand-off – Apache+mod_php
The PHP Hand-off – Apache+suExec[PHP]
The PHP Hand-off – Apache+PHP-FPM
File System Permissions Matter

YES! Do This!!
File System Permissions Matter

666 and 777 settings are both evil!
Fixing File System Permissions
Fix it!! (relative to your web root)
1. Own your files / directories

chmod

find -exec chown eecms.eecms {} ;

2. Make browsing directories for others non-trivial.
find -type d -exec chmod 711 {} ;

3. Make sure the webserver can read it all

711

On What?

Means

Directories You can do anything, others
can ‘list’

644

Files

You can read/write, others
can read

600

Files

You can read/write, others
have no access

find -type f -exec chmod 644 {} ;

4. PHP scripts are for your eyes only
find -type f -name “*.php” -exec chmod 600 {} ;

NOTE – config.php and database.php are 600! This is good!
NOTE2 – This only works in a PHP-FPM or suPHP world!
NOTE3 –You may need root level assistance to achieve this
The Need For Caching
• Any DB (or file) based CMS introduces complexity
• More complexity means slower page loads

• A single sub-optimal query can kill a site’s performance
• This all sounds pretty dire 
• More queries does not necessarily mean slower page loads!
• Developers need strike a balance between speed, complexity and
maintainability
• Caching can fix it!!
ExpressionEngine Built-in Caching
• Tag Caching
• Caches tag results (to disk)

• Good for pinpointing bad tags/queries and caching them on the spot
• BUT… The 1st instance of the tag will still be slower

• Template Caching
• Caches entire templates (to disk)

• Dynamic Channel Query Caching
• Caches parts of the channel entries (to disk)
• MySQL’s query cache can help here too

• Query Disabling
• More a pruning measure than caching but still effective
Benchmarking Caching Strategies
• “Agile Records” default install
• 100,000 additional news entries (thanks EllisLab!)
• This is an atypically sized ExpressionEngine site (i.e. BIG)
• We’re trying to push the limits!

• Dell R410 (What equates to our EEP-400 offering)
• Dual E5620 quad core CPUs
• 16GB RAM
• 4 x 147GB 15k SAS drives (RAID 10)

• Software Snapshot
• ExpressionEngine 2.7.0
• CentOS 6.4 (64-bit)
• PHP 5.3.24 (5.4 would have been faster)
• Percona Server 5.5
Cache
“warming” in
progress

Transactions /
sec go from 7 to
~80!
A warm cache
makes a big
difference 
Avg. response
time goes from
~8 to ~0!

Concurrency = 30
900 Random URLs
200
concurrent
users @ ~260 physical
Server’s
trans/sec
resources
exhausted 
Nexcess’ Memcache Patch
• Replaces the on-disk portions of ExpressionEngine’s default caching
with memcache
• No code (customer) code changes needed

• Memcache is an in-memory data store
• Key/value based
• Very fast compared to disk

• Avoids file-system IO limitations
• File locking
• Read/write performance

• Created as a stop-gap on higher traffic sites
• Open source – BUT BE CAREFUL!
~8 transactions /
sec faster

Concurrency = 30
900 Random URLs
Much better high
traffic performance!
Causing Effect’s Cache Plug-in
• Created by Aaron Waldon of Causing Effect
• Very flexible cache addition to ExpressionEngine
• Memory or file-based back-end, with failover
• Leverage fast memory caching
• Fall back to file based caching if any daemon fails

• Full static page capability
• Good for OMG events

• Benchmarks are done using the full page static caching
• Given EE’s flexible nature this was the easiest/best test
CE Cache’s static
caching hits ~100. EE’s template
caching was ~80
trans/sec

Our patch raised
this to ~88

Concurrency = 30
900 Random URLs
Aaron’s hits a
pretty
consistent 500
trans/sec

Our patch
peaked @ ~360
trans/sec
Caching the Caches With Varnish
• Varnish is a caching HTTP reverse proxy (i.e. web application
accelerator)

• Varnish caches entire rendered pages in memory
• Once a page is in Varnish it bypasses:
• The webserver (Apache)
• The web application (PHP/MySQL)
• And everything else lower in the stack!

• Application level caching still matters!
• Cache misses hit the application

• Using Varnish with even brief (1 minute or less) timeouts can still help

• SO JUST TURN IT ON FOR MY SITE ALREADY!!
Varnish is ~550 

All previous were
~80 – 100
trans/sec

Concurrency = 30
900 Random URLs
Varnish @ ~1000 trans/sec
CE Cache Static Files
The Nexcess patch
EE template caching
Pushing ExpressionEngine Further
• Pitfalls to avoid
• File synchronization
• File-based caching causes flock() contention

• Clustering
•
•
•
•

Load balanced over many web application servers
Unified in-memory caching with dedicated caching server
Varnish front-end servers
DB clustering

• THE FOG…
• Hybrid cloud solution
• Provides on-demand expansion where it counts
• Offload PHP-FPM to the Fog
Final Thoughts
• Detroit is not so bad. Lots of inbound Canadian and outbound U.S.
merchandise passes through it at least!
• Don’t build your house on a rock pillar in the ocean
• 666 & 777 are evil
• PHP-FPM is the way to go
• Use Percona Server over MySQL and InnoDB on high traffic sites
• Caching is good – decide which works for you
• Choose the fastest cache mechanism that makes sense
• In memory caches are almost always the best option

• Varnish is good but requires TLC

• The sky is the limit when it comes to ExpressionEngine performance!
More Information
• Our github repository is at:

https://github.com/nexcess
• Our new performance whitepaper is available at:
http://www.nexcess.net/eecms-best-practices-whitepaper
Thank You!
Questions?
Chris Wells – clwells@nexcess.net

EECI 2013 - ExpressionEngine Performance & Optimization - Laying a Solid Foundation

  • 1.
    ExpressionEngine Performance & Optimization: Layinga Solid Foundation (& Caching) CHR IS WE L L S – CE O – NE XCE SS .NE T L LC
  • 2.
    The Ambassador Bridge(USA/Canada) • Completed in 1929 (Longest suspension bridge worldwide) • Busiest international trade crossing in North America! • Carries 25% of all US/Canadian merchandise trade • Drive south to go north? • Owned by a private citizen !?
  • 3.
    The Ambassador Bridge(USA/Canada) The DetroitWindsor tunnel Detroit, MI USA is cool too! Windsor, Ontario Canada
  • 4.
    Today’s Topics • Whyis the foundation important? • MySQL choices • PHP & the Apache/PHP hand-off • Caching overview • ExpressionEngine internal caching • ExpressionEngine (modified) template caching • Caching with CE cache • Caching the caches with Varnish • Clustering ExpressionEngine
  • 5.
    Perspective • Nexcess isa 13 year old company • 90 employees (customer service being the largest contingent) • Staff of 48 in support/customer service (all U.S./Michigan based) • Wholly owned data centers in the U.S. • Partner data centers in the U.K. and Australia • For better or worse we are not designers • Hosting company / infrastructure view of the system • We often are developers but again with a view that differs from web application developers
  • 6.
    Why Care AboutThe Foundation? • Security, Stability, Performance • Proper security/stability/performance permeates to the bare metal • Do what you can with the access you have, ask questions about what is out of reach • Assume nothing
  • 7.
    The Foundation • Linux •Reboot-less updates (including on the kernel level) • Apache (of Nginx) • MySQL (Percona Server) • PHP • ExpressionEngine
  • 8.
    MySQL (For VeryHigh Traffic Sites) • Don’t use MySQL! • Use Percona Server! • Percona Server is a simple MySQL replacement • Percona is much better under stress • Percona has a very good implementation of InnoDB tables (called XtraDB) • Use InnoDB! • Row (instead of table) level locking • Tune InnoDB for your CPU / RAM configuration – it matters! • A sample my.cnf is on our github account: https://github.com/nexcess/ee-whitepaperv1-configs • MariaDB is an alternative as well although we haven’t benchmarked it (yet)
  • 9.
    PHP Basics • UseAPC (as an opcode cache only) • apc.shm_size • apc.num_files_hint • apc_stat = 256M (at least) = 10000 (at least) = 0 (for production) • Bump PHP’s memory_limit (128M works well) • Turn off open_basedir • Leaving it on kills the realpath cache • Security concerns can be handled with tighter file system security • Use a recent version of PHP (5.4.x+) • 5.4.x is fast! • 5.4.x has a much lower memory footprint than prior versions • Invoke PHP in the right way
  • 10.
    The PHP Hand-off •This handoff is critical for both security and performance • Apache + mod_php • Runs as webserver user • Unified mega-process • Apache + suPHP / phpSuEXEC • Runs as you • Expensive to create • Apache + PHP-FPM • Runs as you • Cheap to use • Scales more efficiently than all of the above
  • 11.
    The PHP Hand-off– Apache+mod_php
  • 12.
    The PHP Hand-off– Apache+suExec[PHP]
  • 13.
    The PHP Hand-off– Apache+PHP-FPM
  • 14.
    File System PermissionsMatter YES! Do This!!
  • 15.
    File System PermissionsMatter 666 and 777 settings are both evil!
  • 16.
    Fixing File SystemPermissions Fix it!! (relative to your web root) 1. Own your files / directories chmod find -exec chown eecms.eecms {} ; 2. Make browsing directories for others non-trivial. find -type d -exec chmod 711 {} ; 3. Make sure the webserver can read it all 711 On What? Means Directories You can do anything, others can ‘list’ 644 Files You can read/write, others can read 600 Files You can read/write, others have no access find -type f -exec chmod 644 {} ; 4. PHP scripts are for your eyes only find -type f -name “*.php” -exec chmod 600 {} ; NOTE – config.php and database.php are 600! This is good! NOTE2 – This only works in a PHP-FPM or suPHP world! NOTE3 –You may need root level assistance to achieve this
  • 17.
    The Need ForCaching • Any DB (or file) based CMS introduces complexity • More complexity means slower page loads • A single sub-optimal query can kill a site’s performance • This all sounds pretty dire  • More queries does not necessarily mean slower page loads! • Developers need strike a balance between speed, complexity and maintainability • Caching can fix it!!
  • 18.
    ExpressionEngine Built-in Caching •Tag Caching • Caches tag results (to disk) • Good for pinpointing bad tags/queries and caching them on the spot • BUT… The 1st instance of the tag will still be slower • Template Caching • Caches entire templates (to disk) • Dynamic Channel Query Caching • Caches parts of the channel entries (to disk) • MySQL’s query cache can help here too • Query Disabling • More a pruning measure than caching but still effective
  • 19.
    Benchmarking Caching Strategies •“Agile Records” default install • 100,000 additional news entries (thanks EllisLab!) • This is an atypically sized ExpressionEngine site (i.e. BIG) • We’re trying to push the limits! • Dell R410 (What equates to our EEP-400 offering) • Dual E5620 quad core CPUs • 16GB RAM • 4 x 147GB 15k SAS drives (RAID 10) • Software Snapshot • ExpressionEngine 2.7.0 • CentOS 6.4 (64-bit) • PHP 5.3.24 (5.4 would have been faster) • Percona Server 5.5
  • 20.
    Cache “warming” in progress Transactions / secgo from 7 to ~80! A warm cache makes a big difference  Avg. response time goes from ~8 to ~0! Concurrency = 30 900 Random URLs
  • 21.
    200 concurrent users @ ~260physical Server’s trans/sec resources exhausted 
  • 22.
    Nexcess’ Memcache Patch •Replaces the on-disk portions of ExpressionEngine’s default caching with memcache • No code (customer) code changes needed • Memcache is an in-memory data store • Key/value based • Very fast compared to disk • Avoids file-system IO limitations • File locking • Read/write performance • Created as a stop-gap on higher traffic sites • Open source – BUT BE CAREFUL!
  • 23.
    ~8 transactions / secfaster Concurrency = 30 900 Random URLs
  • 24.
  • 25.
    Causing Effect’s CachePlug-in • Created by Aaron Waldon of Causing Effect • Very flexible cache addition to ExpressionEngine • Memory or file-based back-end, with failover • Leverage fast memory caching • Fall back to file based caching if any daemon fails • Full static page capability • Good for OMG events • Benchmarks are done using the full page static caching • Given EE’s flexible nature this was the easiest/best test
  • 26.
    CE Cache’s static cachinghits ~100. EE’s template caching was ~80 trans/sec Our patch raised this to ~88 Concurrency = 30 900 Random URLs
  • 27.
    Aaron’s hits a pretty consistent500 trans/sec Our patch peaked @ ~360 trans/sec
  • 28.
    Caching the CachesWith Varnish • Varnish is a caching HTTP reverse proxy (i.e. web application accelerator) • Varnish caches entire rendered pages in memory • Once a page is in Varnish it bypasses: • The webserver (Apache) • The web application (PHP/MySQL) • And everything else lower in the stack! • Application level caching still matters! • Cache misses hit the application • Using Varnish with even brief (1 minute or less) timeouts can still help • SO JUST TURN IT ON FOR MY SITE ALREADY!!
  • 30.
    Varnish is ~550 All previous were ~80 – 100 trans/sec Concurrency = 30 900 Random URLs
  • 31.
    Varnish @ ~1000trans/sec CE Cache Static Files The Nexcess patch EE template caching
  • 32.
    Pushing ExpressionEngine Further •Pitfalls to avoid • File synchronization • File-based caching causes flock() contention • Clustering • • • • Load balanced over many web application servers Unified in-memory caching with dedicated caching server Varnish front-end servers DB clustering • THE FOG… • Hybrid cloud solution • Provides on-demand expansion where it counts • Offload PHP-FPM to the Fog
  • 33.
    Final Thoughts • Detroitis not so bad. Lots of inbound Canadian and outbound U.S. merchandise passes through it at least! • Don’t build your house on a rock pillar in the ocean • 666 & 777 are evil • PHP-FPM is the way to go • Use Percona Server over MySQL and InnoDB on high traffic sites • Caching is good – decide which works for you • Choose the fastest cache mechanism that makes sense • In memory caches are almost always the best option • Varnish is good but requires TLC • The sky is the limit when it comes to ExpressionEngine performance!
  • 34.
    More Information • Ourgithub repository is at: https://github.com/nexcess • Our new performance whitepaper is available at: http://www.nexcess.net/eecms-best-practices-whitepaper
  • 35.
    Thank You! Questions? Chris Wells– clwells@nexcess.net