Strategies and Tips for Building
Enterprise Drupal Applications
Mack Hardy, Dave Tarc, Damien Norris
October 5th, 2013
Pac...
Treat it like a Product
http://affinitybridge.com
● Fixed functionality per release
● Repeatable deployment
● Documentatio...
Application Components
http://affinitybridge.com
What do we mean by Enterprise?
http://affinitybridge.com
● Core to the operations of an organization
● Always changing - e...
Community KnowledgeKeeper
Community KnowledgeKeeper
11 Instances in production
160+ core & contrib modules enabled on larger
installs
40+ custom fea...
Release Management
Keeping the cats herded
http://affinitybridge.com
Version control and releases
● Git for the profile and custom code
● Drush make for fetching
Core+Contrib+Patches
● Branch...
Semantic Versioning
Use Semantic Versioning (http://semver.org)
##.##.##
major #.minor #.point #
Update major # when publi...
● Tag master when a release is ready and
deploy it.
● Distribution is profile managed in a make file,
allows for a rebuild...
● Behat / Mink / Travis CI for automated testing
● Tests run on merge to develop branch
● Test results integrated to chat ...
Building
Now, lets make Drupal play nice
http://affinitybridge.com
Choose a small group of Drupal
modules to build around.
Choose wisely, because
You will have to commit to them.
You will b...
Drupal Base Modules
Features
Ctools
Views
Display Suite
Field Group
Renderable Elements
Context
Omega
Token
Libraries
Enti...
Building with Features
Features enabled functionality to be grouped
together and enabled/disabled.
Features module is almo...
Cascading Features with Field Group
Feature A is required.
Feature B is optional, and requires Feature A.
Feature A implem...
Display Suite with
Renderable Elements
Display Suite allows for configuration of layouts
on content types (as well as othe...
Customized options across installs
Use taxonomy term reference fields if fields
need to have different options across inst...
Search API & Facet API
Using Solr as a backend to return search
results, Search API and Facet API can be used
to build ext...
Pain Points - Role rid’s
Never refer to roles in Features (beyond the
exported role itself.
All (?) role-based operations ...
Pain Points - Default Images
When features exports an image filefield with a
default image, the file fid is what is export...
Pain Points - Update Hooks
Update hooks work great for core and contrib
modules that need to pass updates
downstream to us...
Pain Points - Feature Install Hooks
Feature install hooks are problematic in some
cases:
- Creating taxonomy terms in a vo...
Making updates - Drush scripts
Drush scripts which use EntityFieldQuery and
Entity API for loading, modifying, and saving
...
Making changes
If you really have to, it may be necessary to
change a core component.
For us, OpenLayers was getting more
...
Use the Drupal API
Be wary of adding new modules that offer easy
solutions to problems that can be done easily
using the c...
Rules
Rules offers great automatic functionality that
non-coders can configure. It’s great for some
types of Drupal distro...
Field Permissions
Field Permissions module allows you to assign
view and edit permissions for each field.
Hiding fields on...
Editable Content
Editablefields module works well in some
situations, but is delicate.
Edit module works very well in a va...
Custom Hooks
Create custom hooks within your distro for
greater power
module_invoke_all($hook)
We defined custom hooks for...
Keep Things Impeccable
Use consideration when setting up filefield
paths (consider filefield paths module)
Adopt a standar...
Caching and Performance
we built it, but now they want to go fast too, with scads of data!
http://affinitybridge.com
Caching and performance
on a single Debian Linux server using:
● PHP under FastCGI
● APC: PHP Op-code cache
● Memcached: R...
PHP with FastCGI
Default php5 packages for Linux run using the
Apache module, mod_php. This is problematic:
● All requests...
FastCGI - just like it sounds
CGI: The original, old school web
application spec allows a webserver to
hand off execution ...
FastCGI Other Benefits
● Using SuExec, PHP runs as the correct user!
No more permissions problems or chown
scripts!
● Easy...
APC: Op-code Cache
● PHP files are source code and therefore
interpreted by default.
So, each time the .php file is loaded...
APC: Op-code Cache
APC stores the compiled version in a cache,
greatly speeding up PHP. Instant speed up.
● It can also be...
Memcached : Drupal RAM cache
● Stores Drupal Cache tables in memory
● makes sites Extremely fast
● not just for Anonymous ...
Memcached : HOWTO (1)
● Install “memcached” server (not
“memcachedb” server)
● Get the PHP beta 3.0.8 “memcache” PECL
exte...
Memcached : HOWTO (2)
● Install and enable the memcache Drupal
module.
● Add these magic beans to settings.php to
actually...
XSendFile for private files
● Private files must be served using PHP, so
that Drupal can determine it’s allowed to
serve t...
XSendFile - How it works
Instead of actually opening and outputting the
file, PHP does everything right up to and
includin...
XSendFile - How it works (2)
Then, right when it’s time to open and output the file’s contents, PHP
just exits abruptly in...
XSendFile - Howto
● Download, compile, and enable the XSendFile apache module from
the source code. You will need Apache d...
MySQL tuning with MySQLTuner
Default MySQL settings are pretty gutless.
MySQL will adjust to its tiny cell, but you can co...
Varnish 3 - Front Side HTTP Cache
http://affinitybridge.com
Finally, Varnish is a HTTP Accelerator / Proxy
server that can...
Varnish 3
http://affinitybridge.com
● Unfortunately, it cannot cache content for logged in users, but it can
still serve t...
http://affinitybridge.com
Questions?
Upcoming SlideShare
Loading in …5
×

Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

7,487 views
7,381 views

Published on

Mack Hardy, Dave Tarc, Damien Norris of Affinity Bridge presenting at Pacific Northwest Drupal Summit in Vancouver, October 5th, 2013. The presentation walks through management of releases, deployment strategies and build strategies with drupal features, git, and make files. Performance and caching is also covered, as well as specific tips and tricks for configuring apache and managing private files.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,487
On SlideShare
0
From Embeds
0
Number of Embeds
258
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

  1. 1. Strategies and Tips for Building Enterprise Drupal Applications Mack Hardy, Dave Tarc, Damien Norris October 5th, 2013 Pacific Northwest Drupal Summit
  2. 2. Treat it like a Product http://affinitybridge.com ● Fixed functionality per release ● Repeatable deployment ● Documentation (process, not just end-user). ● QA and Testing (automated and manual)
  3. 3. Application Components http://affinitybridge.com
  4. 4. What do we mean by Enterprise? http://affinitybridge.com ● Core to the operations of an organization ● Always changing - even post launch ● Complex business rules ● Reports ● Data Exports ● Audit Trails ● More Secure ● More Available
  5. 5. Community KnowledgeKeeper
  6. 6. Community KnowledgeKeeper 11 Instances in production 160+ core & contrib modules enabled on larger installs 40+ custom features enabled on larger installs 27 content types 12 taxonomy vocabularies 10 custom user roles
  7. 7. Release Management Keeping the cats herded http://affinitybridge.com
  8. 8. Version control and releases ● Git for the profile and custom code ● Drush make for fetching Core+Contrib+Patches ● Branch-per-Feature development workflow ● Travis-CI and manual testing on stage server ● Careful of dev vs. stage/prod differences (e.g. memory management OSX vs. Linux) http://affinitybridge.com
  9. 9. Semantic Versioning Use Semantic Versioning (http://semver.org) ##.##.## major #.minor #.point # Update major # when public API changes Update minor # on deploys of new functionality (weekly or as needed) Update point # for hotfixes Show version to privileged users on every page
  10. 10. ● Tag master when a release is ready and deploy it. ● Distribution is profile managed in a make file, allows for a rebuild on deploy ● Features are reverted on deploy ● Git Branch Per Feature - See session Sunday Oct 6th, 11:40 “Git Branch-per-Feature” Deployment Process http://affinitybridge.com
  11. 11. ● Behat / Mink / Travis CI for automated testing ● Tests run on merge to develop branch ● Test results integrated to chat client, so devs see the failures ● Limited test coverage, build a test for each new function ● Smith’s presentation “Behaviour Testing and Continuous Integration with Drupal” Sunday Oct 6th, 3pm Automated Testing http://affinitybridge.com
  12. 12. Building Now, lets make Drupal play nice http://affinitybridge.com
  13. 13. Choose a small group of Drupal modules to build around. Choose wisely, because You will have to commit to them. You will be getting heavy use out of them, and pushing their boundaries.
  14. 14. Drupal Base Modules Features Ctools Views Display Suite Field Group Renderable Elements Context Omega Token Libraries Entity API Entity Reference Organic Groups Date Search API Facet API Media Geofield
  15. 15. Building with Features Features enabled functionality to be grouped together and enabled/disabled. Features module is almost required in order to build distros. Features module does have some gaps and will provide some headaches. Features can cascade (depend on each other), which allows for rich options if done well.
  16. 16. Cascading Features with Field Group Feature A is required. Feature B is optional, and requires Feature A. Feature A implements content type Thing. In Order for Feature B to safely put fields on Thing (edit form and full view): - Fields must be placed in field groups - Fields and field groups exported in Feature B - Display Suite settings exported in Feature A
  17. 17. Display Suite with Renderable Elements Display Suite allows for configuration of layouts on content types (as well as other entities) Renderable Elements allows any type of form to be registered & subsequently configured using Display Suite With Omega providing responsive design, this allows for a very consistent look across pages, nodes & edit forms.
  18. 18. Customized options across installs Use taxonomy term reference fields if fields need to have different options across installs. Taxonomy terms are full entities, fieldable and customizable with Display Suite, and all per vocabulary. Depending on the feature, adding a term reference to a term in a parent vocabulary can often work a lot better than using a hierarchical vocabulary
  19. 19. Search API & Facet API Using Solr as a backend to return search results, Search API and Facet API can be used to build extremely customized search pages. Once search results are returned, it is easy to do other things with them, such as export them in a CSV or KML Improve the distro by leveraging use of Search API. Building single-use or inflexible reports with views does not do much for the distro.
  20. 20. Pain Points - Role rid’s Never refer to roles in Features (beyond the exported role itself. All (?) role-based operations work on role rid, which can be different across sites. Views access should never be configured by role. User reference fields should never filter by role. Use permissions instead.
  21. 21. Pain Points - Default Images When features exports an image filefield with a default image, the file fid is what is exported. File fid’s will be different across servers Updating the file fid on each site will override the feature, and future deploys will wipe it In order to have default images which will work across installs, a core patch is required.
  22. 22. Pain Points - Update Hooks Update hooks work great for core and contrib modules that need to pass updates downstream to users Update hooks are a poor choice for managed distros: - DB Updates should be done before clearing cache or updating features
  23. 23. Pain Points - Feature Install Hooks Feature install hooks are problematic in some cases: - Creating taxonomy terms in a vocabulary exported by the feature will not work as the vocabulary will not exist when the hook is run There might be better options in Features 2.x
  24. 24. Making updates - Drush scripts Drush scripts which use EntityFieldQuery and Entity API for loading, modifying, and saving content work great. Drush scripts can be run multiple times. No need to fiddle around with database values to get an update to run again. Drush eval calls also very powerful: drush eval ‘my_function($nid);’
  25. 25. Making changes If you really have to, it may be necessary to change a core component. For us, OpenLayers was getting more cumbersome to use as the system got bigger. We decided to update to Leaflet. Large component changes like this can be time- consuming, messy and a distraction from client desires.
  26. 26. Use the Drupal API Be wary of adding new modules that offer easy solutions to problems that can be done easily using the core hooks. These helper modules will always have a cost. Sometimes that cost is worth it, but be wary of adding stuff you really don’t need.
  27. 27. Rules Rules offers great automatic functionality that non-coders can configure. It’s great for some types of Drupal distros. If your distro is making extensive use of the core hooks, logic implemented via rules can be more challenging to debug. Rules is just implementing hooks that could be implemented directly.
  28. 28. Field Permissions Field Permissions module allows you to assign view and edit permissions for each field. Hiding fields on node form and node view ends up being pretty easy and clean to do in code. hook_form_alter() { $form[<FIELD_NAME>][‘#access’] = 0; hook_node_view() { unset($node->content[<FIELD_NAME>]);
  29. 29. Editable Content Editablefields module works well in some situations, but is delicate. Edit module works very well in a variety of cases but still may not solve the problem. Custom forms built using core Form API put into Display Suite code fields end up working pretty well
  30. 30. Custom Hooks Create custom hooks within your distro for greater power module_invoke_all($hook) We defined custom hooks for: - Site Settings page - Site wide Search API index Features could implement these hooks to include variables on the settings page or fields for search indexing
  31. 31. Keep Things Impeccable Use consideration when setting up filefield paths (consider filefield paths module) Adopt a standardized naming system Adopt a standard module dir layout for include files and drush scripts Categorize your features
  32. 32. Caching and Performance we built it, but now they want to go fast too, with scads of data! http://affinitybridge.com
  33. 33. Caching and performance on a single Debian Linux server using: ● PHP under FastCGI ● APC: PHP Op-code cache ● Memcached: RAM Caching for Drupal ● XSendfile: for private file serving ● MySQL tuning with mysqltuner ● Varnish 3 front-side cache Moar? This stack can and should be clustered. http://affinitybridge.com
  34. 34. PHP with FastCGI Default php5 packages for Linux run using the Apache module, mod_php. This is problematic: ● All requests (even ones for CSS, Images, Text files, anything not . php) are handled by an Apache process with PHP loaded in its memory. This is an inefficient use of memory and in our experience slows things down. ● PHP is run as the Apache user (apache2 or www-data). That user is usually different from the user who owns the files. http://affinitybridge.com
  35. 35. FastCGI - just like it sounds CGI: The original, old school web application spec allows a webserver to hand off execution to another program. Environment variables hold values to be passed between the two. Still used! FastCGI runs a pool of ready-to-go PHP processes and Apache just hands off any requests for a .php to it. It’s the old fashioned way but is really really fast! http://affinitybridge.com
  36. 36. FastCGI Other Benefits ● Using SuExec, PHP runs as the correct user! No more permissions problems or chown scripts! ● Easy to compile different versions of PHP and run them side by side on the same server. For Apache < 2.4, use mod_fcgid for FastCGI. For Apache 2.4+, try PHP-FPM, an improved implementation. http://affinitybridge.com
  37. 37. APC: Op-code Cache ● PHP files are source code and therefore interpreted by default. So, each time the .php file is loaded for a page view, the server has to compile the source code on the fly and this is repeated over and over. http://affinitybridge.com
  38. 38. APC: Op-code Cache APC stores the compiled version in a cache, greatly speeding up PHP. Instant speed up. ● It can also be used for Drupal caching, or as a session handler, but don’t. ● It can be buggy with certain version combinations. (Debian 6 / Ubuntu 12.04.LTS with PHP 5.3. Try leaving apc.shm_size=32 if you are having problems). ● It was once the official cache of choice, and going to be included in PHP 5.4. Now it looks like Zend OpCache (nee Optimizer) is the new darling. http://affinitybridge.com
  39. 39. Memcached : Drupal RAM cache ● Stores Drupal Cache tables in memory ● makes sites Extremely fast ● not just for Anonymous users - speeds up logged in users too because Drupal does lots of caching all over the place. ● Redis also works great for this (and is persistent to disk if reloading cache is expensive) http://affinitybridge.com
  40. 40. Memcached : HOWTO (1) ● Install “memcached” server (not “memcachedb” server) ● Get the PHP beta 3.0.8 “memcache” PECL extension (not the memcached PECL extension) … and make sure it shows up in a phpinfo(); output. http://affinitybridge.com
  41. 41. Memcached : HOWTO (2) ● Install and enable the memcache Drupal module. ● Add these magic beans to settings.php to actually turn it on: /* Memcache */ $conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc'; $conf['cache_default_class'] = 'MemCacheDrupal'; $conf['cache_class_cache_form'] = 'DrupalDatabaseCache'; $conf['memcache_key_prefix'] = sha1(serialize($databases['default']['default'])); http://affinitybridge.com
  42. 42. XSendFile for private files ● Private files must be served using PHP, so that Drupal can determine it’s allowed to serve them up. But for larger files, this is extremely hard on your server. XSendFile (ported over from LightHTTPd) is a clever way to sidestep this issue using just an Apache module and a HTTP header:
  43. 43. XSendFile - How it works Instead of actually opening and outputting the file, PHP does everything right up to and including outputting any content. It outputs a full set of HTTP headers (including a valid Content-length). It adds a header XSendFile: with the path to the file it wants to serve: XSendFile: /home/demo/private/private.pdf
  44. 44. XSendFile - How it works (2) Then, right when it’s time to open and output the file’s contents, PHP just exits abruptly instead, returning no further output after the HTTP headers. As the (empty) page is returned by Apache, the XSendFile module inspects the HTTP headers and it sees XSendFile: <path> and serves the file’s content with Apache, using the correct content-type of the file. PHP is now only involved for a split second and you’re back to the performance you expect and love with public files. Have your cake and eat it too! Or at least... just stop cake spraying all over the walls, furniture, and computers every time someone downloads a file.
  45. 45. XSendFile - Howto ● Download, compile, and enable the XSendFile apache module from the source code. You will need Apache development packages installed. ● Place the EXACT path to your private files directory in the Apache VirtualHost file: # XSendFile provides fast private file xfer using xsendfile Drupal module. <IfModule mod_xsendfile.c> XSendFile on XSendFilePath /home/mysite/private_files </IfModule> ● Install and enable the Drupal xsendfile module. ● Troubleshoot by looking in your Apache error log. The problem is always that the XSendFilePath is wrong, and it will print the one it’s trying to use in the error log so you can fix it and restart Apache.
  46. 46. MySQL tuning with MySQLTuner Default MySQL settings are pretty gutless. MySQL will adjust to its tiny cell, but you can configure it to have way more memory to play with and that will make it go faster. mysqltuner is an awesome command-line utility for tuning MySQL. It will run and analyze your current database and tell you exactly what settings to increase to improve performance. Beware though, the more resources you give MySQL, the slower your server will boot up at first. On CKK, the CPU load spikes to 15 or 20 after a boot and stays there for about 5 minutes till things settle down. Mysqltuner will keep urging you to turn it up until you run out of juice, so use moderation when following its advice - don’t give it all the RAM. http://affinitybridge.com
  47. 47. Varnish 3 - Front Side HTTP Cache http://affinitybridge.com Finally, Varnish is a HTTP Accelerator / Proxy server that can sit in front of your Apache and serve up entire cached pages from RAM. ● It’s mind-bendingly fast and can handle huge traffic spikes without blinking…. provided the entire site can fit in RAM and all your traffic are Anonymous visitors. ● Varnish takes over answering web requests on port 80, and fetches back-end content (when required) from Apache (either on another port or another IP). e.g. Apache on localhost port 8080, Varnish on public network, port 80.
  48. 48. Varnish 3 http://affinitybridge.com ● Unfortunately, it cannot cache content for logged in users, but it can still serve them static files like Images, CSS and Javascript. So if you have large numbers of users it is probably still worth it. if you only have a few users, then Apache serving those files isn’t your bottleneck. ● Varnish is configured using a compiled language called VCL. It’s pretty confusing at first. We use the FourKitchens Drupal config for varnish and recommend you start with that. ● The Drupal varnish module allows Drupal to clear the varnish cache and allows for integration with cache expiry modules. (Check the Boost module page for a great list of options). The cache must be purged when content changes so that people see the latest and greatest. (Not seeing changes appear is a leading cause of irate content editors..)
  49. 49. http://affinitybridge.com Questions?

×