Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

Strategies and Tips for Building
Enterprise Drupal Applications
Mack Hardy, Dave Tarc, Damien Norris
October 5th, 2013
Pacific Northwest Drupal Summit

Treat it like a Product
http://affinitybridge.com
● Fixed functionality per release
● Repeatable deployment
● Documentation (process, not just end-user).
● QA and Testing (automated and manual)

Application Components

What do we mean by Enterprise?
● Core to the operations of an organization
● Always changing - even post launch
● Complex business rules
● Reports
● Data Exports
● Audit Trails
● More Secure
● More Available

Community KnowledgeKeeper
11 Instances in production
160+ core & contrib modules enabled on larger
installs
40+ custom features enabled on larger installs
27 content types
12 taxonomy vocabularies
10 custom user roles

Release Management
Keeping the cats herded

Version control and releases
● Git for the profile and custom code
● Drush make for fetching
Core+Contrib+Patches
● Branch-per-Feature development workflow
● Travis-CI and manual testing on stage server
● Careful of dev vs. stage/prod differences (e.g.
memory management OSX vs. Linux)

Semantic Versioning
Use Semantic Versioning (http://semver.org)
##.##.##
major #.minor #.point #
Update major # when public API changes
Update minor # on deploys of new
functionality (weekly or as needed)
Update point # for hotfixes
Show version to privileged users on every page

● Tag master when a release is ready and
deploy it.
● Distribution is profile managed in a make file,
allows for a rebuild on deploy
● Features are reverted on deploy
● Git Branch Per Feature - See session Sunday
Oct 6th, 11:40 “Git Branch-per-Feature”
Deployment Process

● Behat / Mink / Travis CI for automated testing
● Tests run on merge to develop branch
● Test results integrated to chat client, so devs
see the failures
● Limited test coverage, build a test for each
new function
● Smith’s presentation “Behaviour Testing and
Continuous Integration with Drupal”
Sunday Oct 6th, 3pm
Automated Testing

Building
Now, lets make Drupal play nice

Choose a small group of Drupal
modules to build around.
Choose wisely, because
You will have to commit to them.
You will be getting heavy use
out of them, and
pushing their boundaries.

Drupal Base Modules
Features
Ctools
Views
Display Suite
Field Group
Renderable Elements
Context
Omega
Token
Libraries
Entity API
Entity Reference
Organic Groups
Date
Search API
Facet API
Media
Geofield

Building with Features
Features enabled functionality to be grouped
together and enabled/disabled.
Features module is almost required in order to
build distros.
Features module does have some gaps and will
provide some headaches.
Features can cascade (depend on each other),
which allows for rich options if done well.

Cascading Features with Field Group
Feature A is required.
Feature B is optional, and requires Feature A.
Feature A implements content type Thing.
In Order for Feature B to safely put fields on
Thing (edit form and full view):
- Fields must be placed in field groups
- Fields and field groups exported in Feature B
- Display Suite settings exported in Feature A

Display Suite with
Renderable Elements
Display Suite allows for configuration of layouts
on content types (as well as other entities)
Renderable Elements allows any type of form
to be registered & subsequently configured
using Display Suite
With Omega providing responsive design, this
allows for a very consistent look across pages,
nodes & edit forms.

Customized options across installs
Use taxonomy term reference fields if fields
need to have different options across installs.
Taxonomy terms are full entities, fieldable and
customizable with Display Suite, and all per
vocabulary.
Depending on the feature, adding a term
reference to a term in a parent vocabulary
can often work a lot better than using a
hierarchical vocabulary

Search API & Facet API
Using Solr as a backend to return search
results, Search API and Facet API can be used
to build extremely customized search pages.
Once search results are returned, it is easy to
do other things with them, such as export
them in a CSV or KML
Improve the distro by leveraging use of Search
API. Building single-use or inflexible reports
with views does not do much for the distro.

Pain Points - Role rid’s
Never refer to roles in Features (beyond the
exported role itself.
All (?) role-based operations work on role rid,
which can be different across sites.
Views access should never be configured by
role.
User reference fields should never filter by
role.
Use permissions instead.

Pain Points - Default Images
When features exports an image filefield with a
default image, the file fid is what is exported.
File fid’s will be different across servers
Updating the file fid on each site will override
the feature, and future deploys will wipe it
In order to have default images which will work
across installs, a core patch is required.

Pain Points - Update Hooks
Update hooks work great for core and contrib
modules that need to pass updates
downstream to users
Update hooks are a poor choice for managed
distros:
- DB Updates should be done before clearing
cache or updating features

Pain Points - Feature Install Hooks
Feature install hooks are problematic in some
cases:
- Creating taxonomy terms in a vocabulary
exported by the feature will not work as the
vocabulary will not exist when the hook is run
There might be better options in Features 2.x

Making updates - Drush scripts
Drush scripts which use EntityFieldQuery and
Entity API for loading, modifying, and saving
content work great.
Drush scripts can be run multiple times. No
need to fiddle around with database values to
get an update to run again.
Drush eval calls also very powerful:
drush eval ‘my_function($nid);’

Making changes
If you really have to, it may be necessary to
change a core component.
For us, OpenLayers was getting more
cumbersome to use as the system got bigger.
We decided to update to Leaflet.
Large component changes like this can be time-
consuming, messy and a distraction from
client desires.

Use the Drupal API
Be wary of adding new modules that offer easy
solutions to problems that can be done easily
using the core hooks.
These helper modules will always have a cost.
Sometimes that cost is worth it, but be wary
of adding stuff you really don’t need.

Rules
Rules offers great automatic functionality that
non-coders can configure. It’s great for some
types of Drupal distros.
If your distro is making extensive use of the
core hooks, logic implemented via rules can
be more challenging to debug.
Rules is just implementing hooks that could be
implemented directly.

Field Permissions
Field Permissions module allows you to assign
view and edit permissions for each field.
Hiding fields on node form and node view ends
up being pretty easy and clean to do in code.
hook_form_alter() {
$form[<FIELD_NAME>][‘#access’] = 0;
hook_node_view() {
unset($node->content[<FIELD_NAME>]);

Editable Content
Editablefields module works well in some
situations, but is delicate.
Edit module works very well in a variety of
cases but still may not solve the problem.
Custom forms built using core Form API put
into Display Suite code fields end up working
pretty well

Custom Hooks
Create custom hooks within your distro for
greater power
module_invoke_all($hook)
We defined custom hooks for:
- Site Settings page
- Site wide Search API index
Features could implement these hooks to
include variables on the settings page or fields
for search indexing

Keep Things Impeccable
Use consideration when setting up filefield
paths (consider filefield paths module)
Adopt a standardized naming system
Adopt a standard module dir layout for include
files and drush scripts
Categorize your features

Caching and Performance
we built it, but now they want to go fast too, with scads of data!

Caching and performance
on a single Debian Linux server using:
● PHP under FastCGI
● APC: PHP Op-code cache
● Memcached: RAM Caching for Drupal
● XSendfile: for private file serving
● MySQL tuning with mysqltuner
● Varnish 3 front-side cache
Moar? This stack can and should be clustered.

PHP with FastCGI
Default php5 packages for Linux run using the
Apache module, mod_php. This is problematic:
● All requests (even ones for CSS, Images, Text files, anything not .
php) are handled by an Apache process with PHP loaded in its
memory. This is an inefficient use of memory and in our experience
slows things down.
● PHP is run as the Apache user (apache2 or www-data). That user is
usually different from the user who owns the files.

FastCGI - just like it sounds
CGI: The original, old school web
application spec allows a webserver to
hand off execution to another program.
Environment variables hold values to be
passed between the two. Still used!
FastCGI runs a pool of ready-to-go PHP
processes and Apache just hands off any
requests for a .php to it. It’s the old
fashioned way but is really really fast!

FastCGI Other Benefits
● Using SuExec, PHP runs as the correct user!
No more permissions problems or chown
scripts!
● Easy to compile different versions of PHP and
run them side by side on the same server.
For Apache < 2.4, use mod_fcgid for FastCGI.
For Apache 2.4+, try PHP-FPM, an improved implementation.

APC: Op-code Cache
● PHP files are source code and therefore
interpreted by default.
So, each time the .php file is loaded for a page
view, the server has to compile the source code
on the fly and this is repeated over and over.

APC: Op-code Cache
APC stores the compiled version in a cache,
greatly speeding up PHP. Instant speed up.
● It can also be used for Drupal caching, or as a session handler, but
don’t.
● It can be buggy with certain version combinations. (Debian 6 /
Ubuntu 12.04.LTS with PHP 5.3. Try leaving apc.shm_size=32 if you
are having problems).
● It was once the official cache of choice, and going to be included in
PHP 5.4. Now it looks like Zend OpCache (nee Optimizer) is the new
darling.

Memcached : Drupal RAM cache
● Stores Drupal Cache tables in memory
● makes sites Extremely fast
● not just for Anonymous users - speeds up
logged in users too because Drupal does lots
of caching all over the place.
● Redis also works great for this (and is
persistent to disk if reloading cache is
expensive)

Memcached : HOWTO (1)
● Install “memcached” server (not
“memcachedb” server)
● Get the PHP beta 3.0.8 “memcache” PECL
extension (not the memcached PECL
extension) … and make sure it shows up in a
phpinfo(); output.

Memcached : HOWTO (2)
● Install and enable the memcache Drupal
module.
● Add these magic beans to settings.php to
actually turn it on:
/* Memcache */
$conf['cache_backends'][] =
'sites/all/modules/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
$conf['memcache_key_prefix'] =
sha1(serialize($databases['default']['default']));

XSendFile for private files
● Private files must be served using PHP, so
that Drupal can determine it’s allowed to
serve them up. But for larger files, this is
extremely hard on your server.
XSendFile (ported over from LightHTTPd) is a
clever way to sidestep this issue using just an
Apache module and a HTTP header:

XSendFile - How it works
Instead of actually opening and outputting the
file, PHP does everything right up to and
including outputting any content.
It outputs a full set of HTTP headers (including a
valid Content-length).
It adds a header XSendFile: with the path to the
file it wants to serve:
XSendFile: /home/demo/private/private.pdf

XSendFile - How it works (2)
Then, right when it’s time to open and output the file’s contents, PHP
just exits abruptly instead, returning no further output after the HTTP
headers.
As the (empty) page is returned by Apache, the XSendFile module
inspects the HTTP headers and it sees XSendFile: <path> and serves the
file’s content with Apache, using the correct content-type of the file.
PHP is now only involved for a split second and you’re back to the
performance you expect and love with public files. Have your cake and
eat it too! Or at least... just stop cake spraying all over the walls,
furniture, and computers every time someone downloads a file.

XSendFile - Howto
● Download, compile, and enable the XSendFile apache module from
the source code. You will need Apache development packages
installed.
● Place the EXACT path to your private files directory in the Apache
VirtualHost file:
# XSendFile provides fast private file xfer using xsendfile Drupal module.
<IfModule mod_xsendfile.c>
XSendFile on
XSendFilePath /home/mysite/private_files
</IfModule>
● Install and enable the Drupal xsendfile module.
● Troubleshoot by looking in your Apache error log. The problem is
always that the XSendFilePath is wrong, and it will print the one it’s
trying to use in the error log so you can fix it and restart Apache.

MySQL tuning with MySQLTuner
Default MySQL settings are pretty gutless.
MySQL will adjust to its tiny cell, but you can configure it to have way
more memory to play with and that will make it go faster.
mysqltuner is an awesome command-line utility for tuning MySQL. It
will run and analyze your current database and tell you exactly what
settings to increase to improve performance.
Beware though, the more resources you give MySQL, the slower your
server will boot up at first. On CKK, the CPU load spikes to 15 or 20
after a boot and stays there for about 5 minutes till things settle down.
Mysqltuner will keep urging you to turn it up until you run out of juice,
so use moderation when following its advice - don’t give it all the RAM.

Varnish 3 - Front Side HTTP Cache
Finally, Varnish is a HTTP Accelerator / Proxy
server that can sit in front of your Apache and
serve up entire cached pages from RAM.
● It’s mind-bendingly fast and can handle huge traffic spikes without
blinking…. provided the entire site can fit in RAM and all your
traffic are Anonymous visitors.
● Varnish takes over answering web requests on port 80, and fetches
back-end content (when required) from Apache (either on another
port or another IP). e.g. Apache on localhost port 8080, Varnish on
public network, port 80.

Varnish 3
● Unfortunately, it cannot cache content for logged in users, but it can
still serve them static files like Images, CSS and Javascript. So if you
have large numbers of users it is probably still worth it. if you only
have a few users, then Apache serving those files isn’t your
bottleneck.
● Varnish is configured using a compiled language called VCL. It’s
pretty confusing at first. We use the FourKitchens Drupal config for
varnish and recommend you start with that.
● The Drupal varnish module allows Drupal to clear the varnish cache
and allows for integration with cache expiry modules. (Check the
Boost module page for a great list of options). The cache must be
purged when content changes so that people see the latest and
greatest. (Not seeing changes appear is a leading cause of irate
content editors..)

Questions?

Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

More Related Content

What's hot

Similar to Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

Recently uploaded

Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013