Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

Uploaded on

Mack Hardy, Dave Tarc, Damien Norris of Affinity Bridge presenting at Pacific Northwest Drupal Summit in Vancouver, October 5th, 2013. The presentation walks through management of releases, …

Mack Hardy, Dave Tarc, Damien Norris of Affinity Bridge presenting at Pacific Northwest Drupal Summit in Vancouver, October 5th, 2013. The presentation walks through management of releases, deployment strategies and build strategies with drupal features, git, and make files. Performance and caching is also covered, as well as specific tips and tricks for configuring apache and managing private files.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Strategies and Tips for Building Enterprise Drupal Applications Mack Hardy, Dave Tarc, Damien Norris October 5th, 2013 Pacific Northwest Drupal Summit
  • 2. Treat it like a Product ● Fixed functionality per release ● Repeatable deployment ● Documentation (process, not just end-user). ● QA and Testing (automated and manual)
  • 3. Application Components
  • 4. What do we mean by Enterprise? ● Core to the operations of an organization ● Always changing - even post launch ● Complex business rules ● Reports ● Data Exports ● Audit Trails ● More Secure ● More Available
  • 5. Community KnowledgeKeeper
  • 6. Community KnowledgeKeeper 11 Instances in production 160+ core & contrib modules enabled on larger installs 40+ custom features enabled on larger installs 27 content types 12 taxonomy vocabularies 10 custom user roles
  • 7. Release Management Keeping the cats herded
  • 8. Version control and releases ● Git for the profile and custom code ● Drush make for fetching Core+Contrib+Patches ● Branch-per-Feature development workflow ● Travis-CI and manual testing on stage server ● Careful of dev vs. stage/prod differences (e.g. memory management OSX vs. Linux)
  • 9. Semantic Versioning Use Semantic Versioning ( ##.##.## major #.minor #.point # Update major # when public API changes Update minor # on deploys of new functionality (weekly or as needed) Update point # for hotfixes Show version to privileged users on every page
  • 10. ● Tag master when a release is ready and deploy it. ● Distribution is profile managed in a make file, allows for a rebuild on deploy ● Features are reverted on deploy ● Git Branch Per Feature - See session Sunday Oct 6th, 11:40 “Git Branch-per-Feature” Deployment Process
  • 11. ● Behat / Mink / Travis CI for automated testing ● Tests run on merge to develop branch ● Test results integrated to chat client, so devs see the failures ● Limited test coverage, build a test for each new function ● Smith’s presentation “Behaviour Testing and Continuous Integration with Drupal” Sunday Oct 6th, 3pm Automated Testing
  • 12. Building Now, lets make Drupal play nice
  • 13. Choose a small group of Drupal modules to build around. Choose wisely, because You will have to commit to them. You will be getting heavy use out of them, and pushing their boundaries.
  • 14. Drupal Base Modules Features Ctools Views Display Suite Field Group Renderable Elements Context Omega Token Libraries Entity API Entity Reference Organic Groups Date Search API Facet API Media Geofield
  • 15. Building with Features Features enabled functionality to be grouped together and enabled/disabled. Features module is almost required in order to build distros. Features module does have some gaps and will provide some headaches. Features can cascade (depend on each other), which allows for rich options if done well.
  • 16. Cascading Features with Field Group Feature A is required. Feature B is optional, and requires Feature A. Feature A implements content type Thing. In Order for Feature B to safely put fields on Thing (edit form and full view): - Fields must be placed in field groups - Fields and field groups exported in Feature B - Display Suite settings exported in Feature A
  • 17. Display Suite with Renderable Elements Display Suite allows for configuration of layouts on content types (as well as other entities) Renderable Elements allows any type of form to be registered & subsequently configured using Display Suite With Omega providing responsive design, this allows for a very consistent look across pages, nodes & edit forms.
  • 18. Customized options across installs Use taxonomy term reference fields if fields need to have different options across installs. Taxonomy terms are full entities, fieldable and customizable with Display Suite, and all per vocabulary. Depending on the feature, adding a term reference to a term in a parent vocabulary can often work a lot better than using a hierarchical vocabulary
  • 19. Search API & Facet API Using Solr as a backend to return search results, Search API and Facet API can be used to build extremely customized search pages. Once search results are returned, it is easy to do other things with them, such as export them in a CSV or KML Improve the distro by leveraging use of Search API. Building single-use or inflexible reports with views does not do much for the distro.
  • 20. Pain Points - Role rid’s Never refer to roles in Features (beyond the exported role itself. All (?) role-based operations work on role rid, which can be different across sites. Views access should never be configured by role. User reference fields should never filter by role. Use permissions instead.
  • 21. Pain Points - Default Images When features exports an image filefield with a default image, the file fid is what is exported. File fid’s will be different across servers Updating the file fid on each site will override the feature, and future deploys will wipe it In order to have default images which will work across installs, a core patch is required.
  • 22. Pain Points - Update Hooks Update hooks work great for core and contrib modules that need to pass updates downstream to users Update hooks are a poor choice for managed distros: - DB Updates should be done before clearing cache or updating features
  • 23. Pain Points - Feature Install Hooks Feature install hooks are problematic in some cases: - Creating taxonomy terms in a vocabulary exported by the feature will not work as the vocabulary will not exist when the hook is run There might be better options in Features 2.x
  • 24. Making updates - Drush scripts Drush scripts which use EntityFieldQuery and Entity API for loading, modifying, and saving content work great. Drush scripts can be run multiple times. No need to fiddle around with database values to get an update to run again. Drush eval calls also very powerful: drush eval ‘my_function($nid);’
  • 25. Making changes If you really have to, it may be necessary to change a core component. For us, OpenLayers was getting more cumbersome to use as the system got bigger. We decided to update to Leaflet. Large component changes like this can be time- consuming, messy and a distraction from client desires.
  • 26. Use the Drupal API Be wary of adding new modules that offer easy solutions to problems that can be done easily using the core hooks. These helper modules will always have a cost. Sometimes that cost is worth it, but be wary of adding stuff you really don’t need.
  • 27. Rules Rules offers great automatic functionality that non-coders can configure. It’s great for some types of Drupal distros. If your distro is making extensive use of the core hooks, logic implemented via rules can be more challenging to debug. Rules is just implementing hooks that could be implemented directly.
  • 28. Field Permissions Field Permissions module allows you to assign view and edit permissions for each field. Hiding fields on node form and node view ends up being pretty easy and clean to do in code. hook_form_alter() { $form[<FIELD_NAME>][‘#access’] = 0; hook_node_view() { unset($node->content[<FIELD_NAME>]);
  • 29. Editable Content Editablefields module works well in some situations, but is delicate. Edit module works very well in a variety of cases but still may not solve the problem. Custom forms built using core Form API put into Display Suite code fields end up working pretty well
  • 30. Custom Hooks Create custom hooks within your distro for greater power module_invoke_all($hook) We defined custom hooks for: - Site Settings page - Site wide Search API index Features could implement these hooks to include variables on the settings page or fields for search indexing
  • 31. Keep Things Impeccable Use consideration when setting up filefield paths (consider filefield paths module) Adopt a standardized naming system Adopt a standard module dir layout for include files and drush scripts Categorize your features
  • 32. Caching and Performance we built it, but now they want to go fast too, with scads of data!
  • 33. Caching and performance on a single Debian Linux server using: ● PHP under FastCGI ● APC: PHP Op-code cache ● Memcached: RAM Caching for Drupal ● XSendfile: for private file serving ● MySQL tuning with mysqltuner ● Varnish 3 front-side cache Moar? This stack can and should be clustered.
  • 34. PHP with FastCGI Default php5 packages for Linux run using the Apache module, mod_php. This is problematic: ● All requests (even ones for CSS, Images, Text files, anything not . php) are handled by an Apache process with PHP loaded in its memory. This is an inefficient use of memory and in our experience slows things down. ● PHP is run as the Apache user (apache2 or www-data). That user is usually different from the user who owns the files.
  • 35. FastCGI - just like it sounds CGI: The original, old school web application spec allows a webserver to hand off execution to another program. Environment variables hold values to be passed between the two. Still used! FastCGI runs a pool of ready-to-go PHP processes and Apache just hands off any requests for a .php to it. It’s the old fashioned way but is really really fast!
  • 36. FastCGI Other Benefits ● Using SuExec, PHP runs as the correct user! No more permissions problems or chown scripts! ● Easy to compile different versions of PHP and run them side by side on the same server. For Apache < 2.4, use mod_fcgid for FastCGI. For Apache 2.4+, try PHP-FPM, an improved implementation.
  • 37. APC: Op-code Cache ● PHP files are source code and therefore interpreted by default. So, each time the .php file is loaded for a page view, the server has to compile the source code on the fly and this is repeated over and over.
  • 38. APC: Op-code Cache APC stores the compiled version in a cache, greatly speeding up PHP. Instant speed up. ● It can also be used for Drupal caching, or as a session handler, but don’t. ● It can be buggy with certain version combinations. (Debian 6 / Ubuntu 12.04.LTS with PHP 5.3. Try leaving apc.shm_size=32 if you are having problems). ● It was once the official cache of choice, and going to be included in PHP 5.4. Now it looks like Zend OpCache (nee Optimizer) is the new darling.
  • 39. Memcached : Drupal RAM cache ● Stores Drupal Cache tables in memory ● makes sites Extremely fast ● not just for Anonymous users - speeds up logged in users too because Drupal does lots of caching all over the place. ● Redis also works great for this (and is persistent to disk if reloading cache is expensive)
  • 40. Memcached : HOWTO (1) ● Install “memcached” server (not “memcachedb” server) ● Get the PHP beta 3.0.8 “memcache” PECL extension (not the memcached PECL extension) … and make sure it shows up in a phpinfo(); output.
  • 41. Memcached : HOWTO (2) ● Install and enable the memcache Drupal module. ● Add these magic beans to settings.php to actually turn it on: /* Memcache */ $conf['cache_backends'][] = 'sites/all/modules/memcache/'; $conf['cache_default_class'] = 'MemCacheDrupal'; $conf['cache_class_cache_form'] = 'DrupalDatabaseCache'; $conf['memcache_key_prefix'] = sha1(serialize($databases['default']['default']));
  • 42. XSendFile for private files ● Private files must be served using PHP, so that Drupal can determine it’s allowed to serve them up. But for larger files, this is extremely hard on your server. XSendFile (ported over from LightHTTPd) is a clever way to sidestep this issue using just an Apache module and a HTTP header:
  • 43. XSendFile - How it works Instead of actually opening and outputting the file, PHP does everything right up to and including outputting any content. It outputs a full set of HTTP headers (including a valid Content-length). It adds a header XSendFile: with the path to the file it wants to serve: XSendFile: /home/demo/private/private.pdf
  • 44. XSendFile - How it works (2) Then, right when it’s time to open and output the file’s contents, PHP just exits abruptly instead, returning no further output after the HTTP headers. As the (empty) page is returned by Apache, the XSendFile module inspects the HTTP headers and it sees XSendFile: <path> and serves the file’s content with Apache, using the correct content-type of the file. PHP is now only involved for a split second and you’re back to the performance you expect and love with public files. Have your cake and eat it too! Or at least... just stop cake spraying all over the walls, furniture, and computers every time someone downloads a file.
  • 45. XSendFile - Howto ● Download, compile, and enable the XSendFile apache module from the source code. You will need Apache development packages installed. ● Place the EXACT path to your private files directory in the Apache VirtualHost file: # XSendFile provides fast private file xfer using xsendfile Drupal module. <IfModule mod_xsendfile.c> XSendFile on XSendFilePath /home/mysite/private_files </IfModule> ● Install and enable the Drupal xsendfile module. ● Troubleshoot by looking in your Apache error log. The problem is always that the XSendFilePath is wrong, and it will print the one it’s trying to use in the error log so you can fix it and restart Apache.
  • 46. MySQL tuning with MySQLTuner Default MySQL settings are pretty gutless. MySQL will adjust to its tiny cell, but you can configure it to have way more memory to play with and that will make it go faster. mysqltuner is an awesome command-line utility for tuning MySQL. It will run and analyze your current database and tell you exactly what settings to increase to improve performance. Beware though, the more resources you give MySQL, the slower your server will boot up at first. On CKK, the CPU load spikes to 15 or 20 after a boot and stays there for about 5 minutes till things settle down. Mysqltuner will keep urging you to turn it up until you run out of juice, so use moderation when following its advice - don’t give it all the RAM.
  • 47. Varnish 3 - Front Side HTTP Cache Finally, Varnish is a HTTP Accelerator / Proxy server that can sit in front of your Apache and serve up entire cached pages from RAM. ● It’s mind-bendingly fast and can handle huge traffic spikes without blinking…. provided the entire site can fit in RAM and all your traffic are Anonymous visitors. ● Varnish takes over answering web requests on port 80, and fetches back-end content (when required) from Apache (either on another port or another IP). e.g. Apache on localhost port 8080, Varnish on public network, port 80.
  • 48. Varnish 3 ● Unfortunately, it cannot cache content for logged in users, but it can still serve them static files like Images, CSS and Javascript. So if you have large numbers of users it is probably still worth it. if you only have a few users, then Apache serving those files isn’t your bottleneck. ● Varnish is configured using a compiled language called VCL. It’s pretty confusing at first. We use the FourKitchens Drupal config for varnish and recommend you start with that. ● The Drupal varnish module allows Drupal to clear the varnish cache and allows for integration with cache expiry modules. (Check the Boost module page for a great list of options). The cache must be purged when content changes so that people see the latest and greatest. (Not seeing changes appear is a leading cause of irate content editors..)
  • 49. Questions?