Drupal Cloud to the rescue? Servers, Files, CDNs and Fun!

  • 538 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
538
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Cloud to the rescue? Servers, Files, CDNs and Fun Questions! Alex McFadyen, CTO drupal.org : a_c_m || drupal.org/user/195063 email : alex@demotix.com twitter : @a_c_m
  • 2. Introduction : Demotix.com Demotix.com A two minute (or less) introduction http://prezi.com/_ugoyxm4qprv/demotixcom-2-minutes/ Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 3. Introduction : Demotix.com Some Demotix numbers ● 1.5million+ page views a month ● Very very long tale of content ● Spikes where we get 100ks views in a single day – Oh how I love Varnish ● 14GB+ database ● 750,000+ nodes, even more terms ● 3-7k new nodes per day All running from 3 “real” servers. 1 Web, 1 DB, 1 Util. (Having moved “out” of the cloud 3 years ago) Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 4. Introduction : Why care? Why not just pay for it to be someone else’s problem?● Control (you NEED it)● Complexity (doesnt fit generic solutions)● Policy● If you can. Do. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 5. Introduction : “The Cloud” To the world“Cloud computing refers to the delivery of computing and storage capacity as a service to a heterogeneous community of end- recipients” - http://en.wikipedia.org/wiki/Cloud_computing To me (a Drupal dev) “The cloud is just a buzz word meaning quick to set up VPSs,perhaps unlimited storage space and even a CDN if your lucky” - Me, just now. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 6. Introduction : “The Cloud” 3 parts we were interested inPower = Virtual Private ServersStorage = Disks. Big ones.Delivery = CDNs and other tricks Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 7. Power : Why VPSs rock@ New servers anytime@ Scale up and down anytime@ Short/no contracts@ Total control (I Y Drush, root access, etc)@ Cheap ! Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 8. Power : Why VPSs dont @ Total control (I Y Drush, root access, etc) “With great power comes great responsibility” - Uncle Ben● You always secretly wanted be a sysadmin right?● You want to be woken up at 3am if it goes wrong?● root@server:~# rm -rf / Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 9. Power : Why VPSs dont @ Location (Where did I just put that data?)● EU has laws about where you can send and store users data.● Your business may not allow for you to store data in the cloud.● Cloud may not be where your users are. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 10. Power : Why VPSs dont @ Slices (How big is your share?)● Most (not all) cloud providers slice servers up into equal cpu/ram/disk space chunks and sell them off.● Some let you add on resources you need, others dont. @ Poor disk IO● Are your disks local or over the network? Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 11. Power : Why VPSs dont @ More servers, more problems● Once your past dedicated use servers (web, db etc) you have to do work on the app to support more servers. See last session “How to sleep without the server-crash-fear (High Availability and High Performance Hosting for Drupal)” @ Support● Do you have any?● How about at 3am? ● On Christmas day Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 12. Power : Why VPSs dont @ Bad neighbours & overselling● Starts out great, then reddit moves in next door. @ Cloud A != Cloud B● Disks, CPU/Ram/Disk ratio, network speed, location, management systems etc Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 13. Power : What to do? @ Benchmarks● Drupal benchmark? Omega8.cc Room today at 4pm.● http://cloudharmony.com cpu/ram/disk/web/db Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 14. Power : What to do? @ Try before you buy● Some companies will give you a free week.● Put as close as you can 1:1 copy of your current site on it and... @ Test, test and test some more● AB / Siege test your caching system.● Instead replay your log files with HTTPERF or Jmeter.● Monitor the results with Munin/Cacti/Newrelic. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 15. Power : Review Why VPS rock Why they dont@ New servers anytime @ Total control@ Scale up and down anytime @ Location@ Short/no contracts @ Slices @ Poor disk IO@ Total control @ More servers, more problems@ Cheap ! @ Support @ Overselling @ Cloud A != Cloud BWhat to do@ Benchmark @ Try before you buy@ Test, test and test some more Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 16. Storage : The problem Storage = /sites/*/files@ Multi web head environment@ Web heads need more space than their slice allows@* LOTS of files in a single dir (10k+) Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 17. Storage : The solutionshttps://www.getpantheon.com/news/inside-pantheon- valhalla-filesystem @ Server with big disk(s) / SAN / EBS + NFS ● Tried, tested, works ● Not very exciting ● Single point of failure ● Depending on your cloud could need a 64GB ram server !! Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 18. Storage : The solutions@ API storage + Fuse (Cloudfuse*, s3fs) ● Theoretically unlimited storage ● CDN with little or no extra effort ● A bit bleeding edge / untested@ Gluster FS / Similar ● Impressive solution ● Makes trade-offs drupal doesnt need ● Complicated Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 19. Delivery : Do CDNs matter?Dev server in Paris YES http://www.dotcom-monitor.com/WebTools/technical-tools-browser-test.aspx Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 20. Delivery : Types of CDN http://wimleers.com/article/key-properties-of-a-cdn@ Push (CloudFiles, S3 etc) ● No redundant traffic ● Complex set up@ Pull (lots) ● Very easy to set up ● Redundant traffic@ DNS (Cloudflare/incapsula) ● Takes over your DNS ● Does more than “just” CDN ● We have not tested Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 21. Conclusions@ The Cloud isnt a magic bullet@ The best benchmark is your own app, with replayed traffic.@ Cloud providers differ a LOT@ Storage can be complicated@ CDNs can give you an easy big win Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 22. What did we do?@ Power : Cloud, Yes … but still in testing ● Using Dediserve.com https://billing.dediserve.com/?affid=148 ● Still re-tuning web heads PHP-FPM vs mod_php. ● Hopefully live next week!@ Storage : NFS ● Simple, tried and tested. ● Using a NFS mount used by all web heads.@ CDN : Yes! CDN pull. ● Easy win ! Very easy to set up. ● Dediserves CDN. ● Also using CDN module to spread assets over several domains (improves front end performance) Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 23. * Lots of files / * Cloud fuse@ Lots of files ... Close to 1M files in a single directory ● Wrote custom drupal 6 module to fix ● As part of the file save hooks, it moved the file to a folder structure based on the filename. ● /sites/default/files/<new_folder>/images/123.jpg ● On drupal.org, but alpha@ Cloudfuse + CDN + Lots of files ● Working proof of concept ● Uses Rackspace cloud files for both storage and CDN ● Requires pre-generation of imagecache ● Not published … yet. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  • 24. Thank you Questions?Alex McFadyen, CTOdrupal.org : a_c_m || drupal.org/user/195063 email : alex@demotix.com twitter : @a_c_m