DCAMP         Cloud to the rescue         How I learned to stop worrying and love the cloud         Drupalcamp Arad 2012 -...
DCAMP         About me                                          I pity the fool                                           ...
DCAMP                                                      1                                  The cloud         Drupalcamp...
DCAMP         The cloud          To the world          “Cloud computing refers to the delivery of computing and storage ca...
DCAMP         The promise          ‣ New           servers anytime          ‣ Scale          up and down anytime          ...
DCAMP         Demotix          ‣A      quick 2 minute introduction          ‣ http://prezi.com/_ugoyxm4qprv/demotixcom-2- ...
DCAMP         Demotix numbers          ‣   1.5million+ page views a month          ‣   Very very long tale of content     ...
DCAMP         Demotix before the cloud                                                                        ‣   1 Web se...
DCAMP         Previous cloud experience             ‣   Started on the cloud                 ‣   Prototype and alpha built...
DCAMP         Why go back?             ‣   Spikes                 ‣   Single story spikes                     ‣   Breaking...
DCAMP         Dream setup                                                                        ‣   1+ Load balancer     ...
DCAMP         Why do it yourself?             Why not pay someone else to figure it out?             ‣ Control        - Saa...
DCAMP                                                    2                               Challenges         Drupalcamp Ara...
DCAMP         #1 - Disks             Slice ratio (How big is your share?)             ‣   Most (but not all) cloud provide...
DCAMP         #2 - Location             Laws             ‣   EU has laws about where you can send and store users data.   ...
DCAMP         #3 - Support             More servers, more problems             ‣   Once your past dedicated use servers (w...
DCAMP         #4 - Total control             You need to be a sysadmin or become one             Servers crash at 3AM     ...
DCAMP         #5 - 1 != 1             Cloud providers differ a LOT             ‣   Different slice rations             ‣  ...
DCAMP                                                     3                           Benchmarks         Drupalcamp Arad 2...
DCAMP         Where to start?                                                                        Benchmarks           ...
DCAMP         DB benchmarks         Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue           21Sunday, Novem...
DCAMP         Disk/IO benchmarks         Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue           22Sunday, ...
DCAMP         Mem/IO benchmarks         Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue           23Sunday, N...
DCAMP         CPU benchmarks         Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue           24Sunday, Nove...
DCAMP         Then?             Make sure they can do what you need             ‣   CDN?             ‣   Storage?         ...
DCAMP                                                    4                                        Storage         Drupalca...
DCAMP         Storage             Storage = sites/*/files             More than 1 web server             The slice ratio fa...
DCAMP         NFS             Server with big data + NFS             Tried, tested, works             Not very exciting   ...
DCAMP         API / Gluster             API storage + Fuse (Cloudfuse, s3fs)             Theoretically unlimited storage  ...
DCAMP                                                     5                                               CDNs         Dru...
DCAMP         The problem with CDNs         Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue           31Sunda...
DCAMP         Types of CDNs             Push (CloudFiles, Amazon S3)             No redundant traffic             Complex s...
DCAMP                                                    6                  What did we do?         Drupalcamp Arad 2012 -...
DCAMP         What did we do?                                                                        We picked Dediserve  ...
DCAMP         What did we do?             NFS Storage             Simple, tried and tested             Using a NFS mount o...
DCAMP         Our current setup                                              ‣   1 Load balancer                          ...
DCAMP                                                     7                 Other issues and findings         Drupalcamp A...
DCAMP         Lots of files             Close to 1M files in a single directory             Wrote custom module to fix this ...
DCAMP         What we found             Puppet is awesome             So is any other build automation system             ...
DCAMP                                                     8                            Conclusions         Drupalcamp Arad...
DCAMP         Conclusions             The Cloud isnt a magic bullet             Cloud providers differ a LOT             B...
DCAMP       Thanks!       Questions?       Alexandru Badiu.       Twitter @voidberg       Web http://ctrlz.ro       Email ...
Upcoming SlideShare
Loading in...5
×

Cloud to the rescue? How I learned to stop worrying and love the cloud

761

Published on

Talk given at Drupalcamp Arad 2012.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
761
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Cloud to the rescue? How I learned to stop worrying and love the cloud

  1. 1. DCAMP Cloud to the rescue How I learned to stop worrying and love the cloud Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 1Sunday, November 18, 12
  2. 2. DCAMP About me I pity the fool who doesn’t use Drupal Hi, I’m Alexandru Badiu. I’m a software engineer, amateur game developer and part time Mister T impersonator. Drupal user for 9 years, board member in Drupal Romania. I work for Demotix. Twitter @voidberg Web ctrlz.ro Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 2Sunday, November 18, 12
  3. 3. DCAMP 1 The cloud Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 3Sunday, November 18, 12
  4. 4. DCAMP The cloud To the world “Cloud computing refers to the delivery of computing and storage capacity as a service to a heterogeneous community of end-recipients” - Wikipedia To us “The cloud is just a buzz word meaning quick to set up VPSs when you need them, perhaps have unlimited storage space and even a CDN if your lucky” - Demotix CTO Web “2.0” Servers - Virtual Private Servers Storage - Lots and lots of data Delivery - CDNs and other tricks Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 4Sunday, November 18, 12
  5. 5. DCAMP The promise ‣ New servers anytime ‣ Scale up and down anytime ‣ Unlimited storage ‣ Pay only for what you use ‣ Short / no contracts ‣ Total control Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 5Sunday, November 18, 12
  6. 6. DCAMP Demotix ‣A quick 2 minute introduction ‣ http://prezi.com/_ugoyxm4qprv/demotixcom-2- minutes/ ‣ Acquired by Corbis a couple of days ago Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 6Sunday, November 18, 12
  7. 7. DCAMP Demotix numbers ‣ 1.5million+ page views a month ‣ Very very long tale of content ‣ Spikes where we get 100ks views in a single day ‣ 14 GB+ database ‣ 1M+ nodes, even more terms (10M+) ‣ 3-5k new nodes per day ‣ 200GB+ files directory ‣ Some folders have over 750K files ‣ 80% unauthenticated traffic ‣ 6k active users Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 7Sunday, November 18, 12
  8. 8. DCAMP Demotix before the cloud ‣ 1 Web server ‣ Varnish ‣ Apache, mod_php ‣ 1 Database server ‣ MySQL ‣ 1 Util server ‣ Memcached ‣ Solr ‣ FTP Server ‣ Beanstalk Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 8Sunday, November 18, 12
  9. 9. DCAMP Previous cloud experience ‣ Started on the cloud ‣ Prototype and alpha built a cloud ‣ Dev environment still on a cloud ‣ Moved off for better support ‣ We needed 24/7 support ‣ Site liked to crash on major Christian holidays ‣ Saw a nice speed bump when we did ‣ Better disk IO ‣ Dedicated hardware Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 9Sunday, November 18, 12
  10. 10. DCAMP Why go back? ‣ Spikes ‣ Single story spikes ‣ Breaking images of the Oslo bomb story went viral ‣ Our firewall (not webserver) went into melt down ‣ Event based spikes ‣ Royal wedding ‣ Elections ‣ Olympics ‣ Dev = Staging = Production ‣ Flexibility Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 10Sunday, November 18, 12
  11. 11. DCAMP Dream setup ‣ 1+ Load balancer ‣ Varnish ‣ 3+ Web servers ‣ Nginx, PHP-FPM ‣ 2+ Database servers ‣ MariaDB ‣ 2+ Util server ‣ Memcached ‣ Solr ‣ FTP Server ‣ Beanstalk Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 11Sunday, November 18, 12
  12. 12. DCAMP Why do it yourself? Why not pay someone else to figure it out? ‣ Control - SaaS is a blackbox in most cases ‣ Complexity - We did not fit generic solutions ‣ If you can do it, go ahead Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 12Sunday, November 18, 12
  13. 13. DCAMP 2 Challenges Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 13Sunday, November 18, 12
  14. 14. DCAMP #1 - Disks Slice ratio (How big is your share?) ‣ Most (but not all) cloud providers slice servers up into equal cpu/ ram/disk space chunks and sell them off. ‣ Some let you add on resources you need, others dont. ‣ Problem for file servers. Poor disk IO ‣ Are your disks local or over the network? Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 14Sunday, November 18, 12
  15. 15. DCAMP #2 - Location Laws ‣ EU has laws about where you can send and store users data. ‣ Your business may not allow for you to store data in the cloud. Users ‣ Cloud may not be where your users are. ‣ CDNs can help. Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 15Sunday, November 18, 12
  16. 16. DCAMP #3 - Support More servers, more problems ‣ Once your past dedicated use servers (web, db etc) you have to do work on the app to support more servers. Help ‣ Can you get any? And how fast? ‣ How about at 3am? ‣ On Christmas day Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 16Sunday, November 18, 12
  17. 17. DCAMP #4 - Total control You need to be a sysadmin or become one Servers crash at 3AM ‣ Monitoring systems ‣ Emergency instructions ‣ Disaster recovery Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 17Sunday, November 18, 12
  18. 18. DCAMP #5 - 1 != 1 Cloud providers differ a LOT ‣ Different slice rations ‣ Different bandwidth allocation methods ‣ Different support packages ‣ Very different performance Overselling ‣ Happens a lot Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 18Sunday, November 18, 12
  19. 19. DCAMP 3 Benchmarks Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 19Sunday, November 18, 12
  20. 20. DCAMP Where to start? Benchmarks ‣ Learn to benchmark Drupal ‣ Cloudharmony.com Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 20Sunday, November 18, 12
  21. 21. DCAMP DB benchmarks Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 21Sunday, November 18, 12
  22. 22. DCAMP Disk/IO benchmarks Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 22Sunday, November 18, 12
  23. 23. DCAMP Mem/IO benchmarks Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 23Sunday, November 18, 12
  24. 24. DCAMP CPU benchmarks Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 24Sunday, November 18, 12
  25. 25. DCAMP Then? Make sure they can do what you need ‣ CDN? ‣ Storage? ‣ Good support? ‣ Is it going to break the bank? Try before you buy ‣ If there’s a free testing period, use it ‣ Put as close to a 1:1 copy of your site Hammer it ‣ AB / Siege test the cache system ‣ Replay your log files with HTTPERF or Jmeter ‣ Monitor the results with Munin/Cacti/NewRelic Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 25Sunday, November 18, 12
  26. 26. DCAMP 4 Storage Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 26Sunday, November 18, 12
  27. 27. DCAMP Storage Storage = sites/*/files More than 1 web server The slice ratio fails for file servers Lots of files in a single directory (10k+) https://www.getpantheon.com/news/inside- pantheon-valhalla-filesystem Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 27Sunday, November 18, 12
  28. 28. DCAMP NFS Server with big data + NFS Tried, tested, works Not very exciting Single point of failure Depending on cloud you could need a 64 GB RAM server Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 28Sunday, November 18, 12
  29. 29. DCAMP API / Gluster API storage + Fuse (Cloudfuse, s3fs) Theoretically unlimited storage Easy CDN A bit bleeding edge Gluster FS Impressive Tradeoffs that Drupal doesn’t need Complicated but likely worth it Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 29Sunday, November 18, 12
  30. 30. DCAMP 5 CDNs Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 30Sunday, November 18, 12
  31. 31. DCAMP The problem with CDNs Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 31Sunday, November 18, 12
  32. 32. DCAMP Types of CDNs Push (CloudFiles, Amazon S3) No redundant traffic Complex setup Pull (Lots) Very easy to setup Lots of redundant traffic DNS (Cloudflare / Incapsula) Takes over your DNS Does much more (analytics, protection, optimization) Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 32Sunday, November 18, 12
  33. 33. DCAMP 6 What did we do? Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 33Sunday, November 18, 12
  34. 34. DCAMP What did we do? We picked Dediserve http://dediserve.com Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 34Sunday, November 18, 12
  35. 35. DCAMP What did we do? NFS Storage Simple, tried and tested Using a NFS mount of raid disks used by all webheads CDN Pull Very easy to setup Dediserve’s OnApp CDN Spread assets over several domains (front end performance) Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 35Sunday, November 18, 12
  36. 36. DCAMP Our current setup ‣ 1 Load balancer ‣ Varnish ‣ 4 Web servers ‣ Apache, mod_php ‣ 1 Database servers ‣ MariaDB ‣ 1 Util server ‣ Memcached ‣ Solr ‣ FTP Server ‣ Beanstalk Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 36Sunday, November 18, 12
  37. 37. DCAMP 7 Other issues and findings Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 37Sunday, November 18, 12
  38. 38. DCAMP Lots of files Close to 1M files in a single directory Wrote custom module to fix this Part of the file save hooks it moved the file to a folder structure based on the filename Will be released on d.o Cloudfuse + CDN Working proof of concept Uses Rackspace’s cloud files for both storage and CDN Requires pre-generation of imagecache Not published yet Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 38Sunday, November 18, 12
  39. 39. DCAMP What we found Puppet is awesome So is any other build automation system Spinning up new test or production servers is very very easy Great to setup local dev, stage or merge url_alias is dog slow on InnoDB Removing the language conditions speeds it up Convert term_data’s description to VARCHAR MySQL won’t use in memory tables if you query TEXT fields Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 39Sunday, November 18, 12
  40. 40. DCAMP 8 Conclusions Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 40Sunday, November 18, 12
  41. 41. DCAMP Conclusions The Cloud isnt a magic bullet Cloud providers differ a LOT Benchmarking can help differentiate providers The best benchmark is your own app, with replayed traffic. Storage can be complicated CDNs can give you an easy win Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 41Sunday, November 18, 12
  42. 42. DCAMP Thanks! Questions? Alexandru Badiu. Twitter @voidberg Web http://ctrlz.ro Email andu@ctrlz.ro D.O http://drupal.org/user/8662 Drupalcamp Arad 2012 - Alexandru Badiu - Cloud to the rescue 42Sunday, November 18, 12
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×