A presentation discussing how to run a large-scale Drupal installation using Amazon Web Services (AWS). The final system is capable of serving millions of unique pages, and storing tens of terabytes of data. First presented at DrupalCamp Brighton in January 2015. There is an hour long recording of this presentation at https://www.youtube.com/watch?v=Rh_yBzRpOnk

1. Scaling Drupal on
Amazon Web Services
Tristan Roddis
DrupalCamp Brighton, January 2015

2. What we created: www.qdl.qa
2

3. 3

4. 4

5. 5

6. • Several hundred thousand pages
• Tens of terabytes of data
• Requirement for high throughput
• Requirement for high resilience
6

7. What is Drupal?
7

8. Let’s scale it!
8

9. Let’s not scale it like this!
9

10. Centralised database
10

11. Centralised filesystem
11

12. Avoid single point of failure
12

13. AWS TLAs
• EC2 – Elastic Compute Cloud = virtual servers
• RDS – Relational Database Service = database
• ELB – Elastic Load Balancer = distribute requests
13

14. Therefore…
14

15. RDS has easy redundancy
15

16. So now we have
16

17. Clustered filesystem: GlusterFS
17

18. GlusterFS components
• EBS – Elastic Block Storage
= file storage up to 1Tb
• LVM – Logical Volume Manager
= grow and shrink volumes
• Gluster has storage ‘bricks’
• Cluster appears as single mount point
(NFS, Gluster protocol etc.)
18

19. Distribute servers
• Amazon has different Regions and
Availability Zones
19

20. Use different Availability Zones
20

21. I lied…
21

22. Adding Solr and IIPImage…
22

23. Additional security
• VPC – Virtual Private Cloud = can create your
own network architecture
• Security Groups = firewall rules
• IAM = Identity and Access Management =
restrict AWS roles
23

24. Final setup
24

25. Woah – that’s complicated
25

26. Server provisioning
• Automate everything!
• Use Ansible (or Chef, Puppet, etc.)
26
“If you have to SSH into your servers,
then your automation has failed.”
— Rich Adams

27. 27

28. Pets vs cattle
28
“When one of them gets sick, you shoot
'em in the head and replace 'em with a
new one.”
—Randy Bias, CEO of Cloudscaling

29. Deployment I: Drupal tools
• Drush
• Drush master
• Features
• Strongarm
29

30. Deployment II
• Use Fabric (or Ansible, Capistrano, etc.)
• @runs_once decorator for e.g. drush commands.
30

31. Load Test
• Jmeter
• Blitz.io or similar
31

32. Tune performance
• Tools: Apache Benchmark (ab), Yslow, Google
PageSpeed
• xhprof – remove bottlenecks
• Fix warnings and notices
• Disable database logging
• OpCode cache (Zend Optimizer, APC)
• Minify CSS/JS
• Lazy Loading via AJAX
32

33. 33

34. Add caching
• Varnish
• Redis/memcached for cache tables (can use
Amazon Elasticache)
• CDN (e.g. Amazon Cloudfront)
• Re-run load tests
34

35. [auto]scaling
• AMIs = Amazon Machine Image
• Autoscaling policies
• “pets versus cattle” again
• Problem: servers must have latest Drupal code
• Conflict with Fabric 'push‘ technique
• Interim solution: create an AMI after
each deployment
35

36. Backup
• AMIs - Amazon Machine Image = snapshot
server images
• ELB snapshots
• S3 – Simple Storage Service = REST storage
• Glacier = long-term storage
• We used:
– Snapshots with RDS
– S3 with versioning, plus Glacier for Drupal
assets and image assets
36

37. Monitoring I: Cloudwatch alerts
37

38. Monitoring II: log analysis
• SumoLogic to aggregate logs (or Loggly,
Papertrail, etc.)
38

39. Monitoring III: notifications
• OpsGenie (or PagerDuty or similar)
39

40. Gotchas
• GlusterFS for PHP files – don’t!
• Custom Drupal cache – careful of
uncontrolled growth!
40

41. Tips
• Use “termination protection”
• Enable ELB logging
• “Maintenance 200” module
• Create a “down for maintenance” server
41

42. Conclusion
• 21 instances
• 6 load balancers
• 67 ELB volumes
• Lot of work to set up, but quick to alter
• Easily coped with launch traffic:
– 21k sessions/day
– peak of 350 concurrent users
42

43. Further Reading
• Justin Slattery, "Multiple Region Autoscaling
Drupal in Amazon Web Services"
http://dev.mlsdigital.net/posts/Cloud-Native-
Drupal/
• Rich Adams, "AWS Tips I Wish I'd Known Before
I Started" https://wblinks.com/notes/aws-tips-i-
wish-id-known-before-i-started/
• Laura Hamilton, "Are your servers pets or
cattle?" http://www.lauradhamilton.com/servers-
pets-versus-cattle
43

44. www.cogapp.com/careers

45. Questions?
• tristanr@cogapp.com
45