Cloud to the rescue?  Servers, Files, CDNs and Fun           Questions!      Alex McFadyen, CTO     drupal.org : a_c_m || ...
Introduction : Demotix.com                   Demotix.com  A two minute (or less) introduction   http://prezi.com/_ugoyxm4q...
Introduction : Demotix.com               Some Demotix numbers ●     1.5million+ page views a month ●     Very very long ta...
Introduction : Why care?      Why not just pay for it to be someone else’s                       problem?●    Control (you...
Introduction : “The Cloud”                              To the world“Cloud computing refers to the delivery of computing a...
Introduction : “The Cloud”  3 parts we were interested inPower         = Virtual Private ServersStorage       = Disks. Big...
Power : Why VPSs rock@   New servers anytime@   Scale up and down anytime@   Short/no contracts@   Total control          ...
Power : Why VPSs dont    @ Total control                 (I Y Drush, root access, etc)       “With great power comes great...
Power : Why VPSs dont    @ Location            (Where did I just put that data?)●    EU has laws about where you can send ...
Power : Why VPSs dont    @ Slices        (How big is your share?)●    Most (not all) cloud providers slice servers up into...
Power : Why VPSs dont    @ More servers, more problems●    Once your past dedicated use servers (web, db etc) you    have ...
Power : Why VPSs dont    @ Bad neighbours & overselling●    Starts out great, then reddit moves in next door.    @ Cloud A...
Power : What to do?    @ Benchmarks●    Drupal benchmark? Omega8.cc Room today at 4pm.●    http://cloudharmony.com cpu/ram...
Power : What to do?    @ Try before you buy●    Some companies will give you a free week.●    Put as close as you can 1:1 ...
Power : Review   Why VPS rock                                                    Why they dont@ New servers anytime       ...
Storage : The problem          Storage = /sites/*/files@ Multi web head environment@ Web heads need more space than   thei...
Storage : The solutionshttps://www.getpantheon.com/news/inside-pantheon-                valhalla-filesystem   @ Server wit...
Storage : The solutions@ API storage + Fuse (Cloudfuse*, s3fs)   ●       Theoretically unlimited storage   ●       CDN wit...
Delivery : Do CDNs matter?Dev server in Paris       YES     http://www.dotcom-monitor.com/WebTools/technical-tools-browser...
Delivery : Types of CDN http://wimleers.com/article/key-properties-of-a-cdn@ Push (CloudFiles, S3 etc)     ●         No re...
Conclusions@ The Cloud isnt a magic bullet@ The best benchmark is your own app,   with replayed traffic.@  Cloud providers...
What did we do?@ Power : Cloud, Yes … but still in testing    ●        Using Dediserve.com             https://billing.ded...
* Lots of files / * Cloud fuse@ Lots of files ...   Close to 1M files in a single directory    ●        Wrote custom drupa...
Thank you   Questions?Alex McFadyen, CTOdrupal.org : a_c_m || drupal.org/user/195063    email : alex@demotix.com   twitter...
Upcoming SlideShare
Loading in …5
×

Drupal Cloud to the rescue? Servers, Files, CDNs and Fun!

932 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
932
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Drupal Cloud to the rescue? Servers, Files, CDNs and Fun!

  1. 1. Cloud to the rescue? Servers, Files, CDNs and Fun Questions! Alex McFadyen, CTO drupal.org : a_c_m || drupal.org/user/195063 email : alex@demotix.com twitter : @a_c_m
  2. 2. Introduction : Demotix.com Demotix.com A two minute (or less) introduction http://prezi.com/_ugoyxm4qprv/demotixcom-2-minutes/ Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  3. 3. Introduction : Demotix.com Some Demotix numbers ● 1.5million+ page views a month ● Very very long tale of content ● Spikes where we get 100ks views in a single day – Oh how I love Varnish ● 14GB+ database ● 750,000+ nodes, even more terms ● 3-7k new nodes per day All running from 3 “real” servers. 1 Web, 1 DB, 1 Util. (Having moved “out” of the cloud 3 years ago) Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  4. 4. Introduction : Why care? Why not just pay for it to be someone else’s problem?● Control (you NEED it)● Complexity (doesnt fit generic solutions)● Policy● If you can. Do. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  5. 5. Introduction : “The Cloud” To the world“Cloud computing refers to the delivery of computing and storage capacity as a service to a heterogeneous community of end- recipients” - http://en.wikipedia.org/wiki/Cloud_computing To me (a Drupal dev) “The cloud is just a buzz word meaning quick to set up VPSs,perhaps unlimited storage space and even a CDN if your lucky” - Me, just now. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  6. 6. Introduction : “The Cloud” 3 parts we were interested inPower = Virtual Private ServersStorage = Disks. Big ones.Delivery = CDNs and other tricks Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  7. 7. Power : Why VPSs rock@ New servers anytime@ Scale up and down anytime@ Short/no contracts@ Total control (I Y Drush, root access, etc)@ Cheap ! Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  8. 8. Power : Why VPSs dont @ Total control (I Y Drush, root access, etc) “With great power comes great responsibility” - Uncle Ben● You always secretly wanted be a sysadmin right?● You want to be woken up at 3am if it goes wrong?● root@server:~# rm -rf / Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  9. 9. Power : Why VPSs dont @ Location (Where did I just put that data?)● EU has laws about where you can send and store users data.● Your business may not allow for you to store data in the cloud.● Cloud may not be where your users are. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  10. 10. Power : Why VPSs dont @ Slices (How big is your share?)● Most (not all) cloud providers slice servers up into equal cpu/ram/disk space chunks and sell them off.● Some let you add on resources you need, others dont. @ Poor disk IO● Are your disks local or over the network? Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  11. 11. Power : Why VPSs dont @ More servers, more problems● Once your past dedicated use servers (web, db etc) you have to do work on the app to support more servers. See last session “How to sleep without the server-crash-fear (High Availability and High Performance Hosting for Drupal)” @ Support● Do you have any?● How about at 3am? ● On Christmas day Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  12. 12. Power : Why VPSs dont @ Bad neighbours & overselling● Starts out great, then reddit moves in next door. @ Cloud A != Cloud B● Disks, CPU/Ram/Disk ratio, network speed, location, management systems etc Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  13. 13. Power : What to do? @ Benchmarks● Drupal benchmark? Omega8.cc Room today at 4pm.● http://cloudharmony.com cpu/ram/disk/web/db Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  14. 14. Power : What to do? @ Try before you buy● Some companies will give you a free week.● Put as close as you can 1:1 copy of your current site on it and... @ Test, test and test some more● AB / Siege test your caching system.● Instead replay your log files with HTTPERF or Jmeter.● Monitor the results with Munin/Cacti/Newrelic. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  15. 15. Power : Review Why VPS rock Why they dont@ New servers anytime @ Total control@ Scale up and down anytime @ Location@ Short/no contracts @ Slices @ Poor disk IO@ Total control @ More servers, more problems@ Cheap ! @ Support @ Overselling @ Cloud A != Cloud BWhat to do@ Benchmark @ Try before you buy@ Test, test and test some more Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  16. 16. Storage : The problem Storage = /sites/*/files@ Multi web head environment@ Web heads need more space than their slice allows@* LOTS of files in a single dir (10k+) Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  17. 17. Storage : The solutionshttps://www.getpantheon.com/news/inside-pantheon- valhalla-filesystem @ Server with big disk(s) / SAN / EBS + NFS ● Tried, tested, works ● Not very exciting ● Single point of failure ● Depending on your cloud could need a 64GB ram server !! Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  18. 18. Storage : The solutions@ API storage + Fuse (Cloudfuse*, s3fs) ● Theoretically unlimited storage ● CDN with little or no extra effort ● A bit bleeding edge / untested@ Gluster FS / Similar ● Impressive solution ● Makes trade-offs drupal doesnt need ● Complicated Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  19. 19. Delivery : Do CDNs matter?Dev server in Paris YES http://www.dotcom-monitor.com/WebTools/technical-tools-browser-test.aspx Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  20. 20. Delivery : Types of CDN http://wimleers.com/article/key-properties-of-a-cdn@ Push (CloudFiles, S3 etc) ● No redundant traffic ● Complex set up@ Pull (lots) ● Very easy to set up ● Redundant traffic@ DNS (Cloudflare/incapsula) ● Takes over your DNS ● Does more than “just” CDN ● We have not tested Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  21. 21. Conclusions@ The Cloud isnt a magic bullet@ The best benchmark is your own app, with replayed traffic.@ Cloud providers differ a LOT@ Storage can be complicated@ CDNs can give you an easy big win Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  22. 22. What did we do?@ Power : Cloud, Yes … but still in testing ● Using Dediserve.com https://billing.dediserve.com/?affid=148 ● Still re-tuning web heads PHP-FPM vs mod_php. ● Hopefully live next week!@ Storage : NFS ● Simple, tried and tested. ● Using a NFS mount used by all web heads.@ CDN : Yes! CDN pull. ● Easy win ! Very easy to set up. ● Dediserves CDN. ● Also using CDN module to spread assets over several domains (improves front end performance) Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  23. 23. * Lots of files / * Cloud fuse@ Lots of files ... Close to 1M files in a single directory ● Wrote custom drupal 6 module to fix ● As part of the file save hooks, it moved the file to a folder structure based on the filename. ● /sites/default/files/<new_folder>/images/123.jpg ● On drupal.org, but alpha@ Cloudfuse + CDN + Lots of files ● Working proof of concept ● Uses Rackspace cloud files for both storage and CDN ● Requires pre-generation of imagecache ● Not published … yet. Drupal Developer Days – Barcelona June 16th 2012 ALEX McFADYEN | alex@demotix.com | @a_c_m
  24. 24. Thank you Questions?Alex McFadyen, CTOdrupal.org : a_c_m || drupal.org/user/195063 email : alex@demotix.com twitter : @a_c_m

×