How to build a state-of-the-art rails cluster

15,111 views

Published on

presented at 'ostrava on rails' on 2007-06-22. there exists also a video recording:

http://www.rails.cz/articles/2007/06/27/oor-videos

Published in: Economy & Finance, Technology
6 Comments
19 Likes
Statistics
Notes
No Downloads
Views
Total views
15,111
On SlideShare
0
From Embeds
0
Number of Embeds
83
Actions
Shares
0
Downloads
420
Comments
6
Likes
19
Embeds 0
No embeds

No notes for slide

How to build a state-of-the-art rails cluster

  1. 1. How to build a state-of-the-art rails cluster Tim Lossen Infopark AG, Berlin
  2. 2. You mean ... like this?
  3. 3. System X • Virginia Tech • 1100 x Apple XServe G5 • 2 CPUs each (2,3 GHz) • Infiniband network • 12 Teraflops
  4. 4. How to build a state-of-the-art rails cluster for 100.000 Koruny Tim Lossen Infopark AG, Berlin
  5. 5. 100.000 Koruny = 4700 $ = 3500 €
  6. 6. • Evolving a cluster setup • Scaling • Make or buy? • Discussion
  7. 7. Aims • performance • efficiency • availability • scalability
  8. 8. Performance • pages / second • throughput • 10 Mbit connection = max. about 1 million unique page impressions per day
  9. 9. Efficiency • bang per buck • bang per watt
  10. 10. Availability • uptime • can be expressed in ‚nines‘ • 99% (2 nines) = max. 100 minutes downtime per week • 99,9999% (6 nines) = max. 0,6 seconds downtime per week
  11. 11. Scalability • grow (or shrink) app as needed • ideal: linear with number of machines
  12. 12. Setup 1
  13. 13. Example software • nginx • mongrel cluster • mysql
  14. 14. Ideas • put everything on one machine • use top quality hardware • go for high processor speed
  15. 15. Example hardware • Sun Fire X2200 M2 • 2 x Opteron 2220 (dual core / 2,8 GHz) • 4 GB ram • 250 GB disk • 3290 €
  16. 16. Result • performance ✔ • efficiency • availability • scalability
  17. 17. Setup 2
  18. 18. Ideas • put app and database on separate machines • use cheap commodity hardware • like Google! • go for high number of CPUs
  19. 19. Example hardware • Dell PowerEdge SC 1435 • 2 x Opteron 2212 (dual core / 2 GHz) • 4 GB ram, 160 GB disk • 2 machines = 8 cores • 3340 €
  20. 20. Result • performance ✔ • efficiency ✔ • availability • scalability
  21. 21. Setup 3
  22. 22. Ideas • cluster of identical nodes • loadbalancing • failover • eliminate single points of failure through redundancy
  23. 23. Setup 3a
  24. 24. Virtualization • partition one physical machine into multiple virtual machines • assign dedicated ressources (cpu, ram) to virtual machines • example: xen (linux), zones (solaris)
  25. 25. Why virtualization? • flexibility • simplified deployment • move images between machines • optimization • hardware utilization • tuning
  26. 26. Why virtualization? • isolation • use different operating systems • contain intrusion • multiple environments (staging, production) • but: small performance overhead (ca. 5%)
  27. 27. Setup 3a
  28. 28. Loadbalancer • openBSD • CARP: shared virtual ip, automatic failover • pf: firewall + loadbalancer • openBSD on xen: currently only on ramdisk • better use dedicated hardware instead
  29. 29. Redundant firewall
  30. 30. Database • mysql • master-master replication • caveat: avoid primary key clashes! • other possibilities: • master-slave replication • mysql cluster
  31. 31. Result • performance ✔ • efficiency ✔ • availability ✔ • scalability ?
  32. 32. • Evolving a cluster setup • Scaling • Make or buy? • Discussion
  33. 33. Scaling • not a rails-specific problem • find and eliminate bottlenecks!
  34. 34. Scaling approaches • scaling up • increase power of machines • scaling out • increase number of machines
  35. 35. Scaling up: memory • add more memory • mongrel can be very memory-hungry • useful for caching (memcached) • hint: check crucial.com for low prices
  36. 36. Scaling up: CPU • upgrade to quad-core Opterons • available later this year • same socket • same power / heat envelope • use as drop-in replacement
  37. 37. Scaling out
  38. 38. Scaling out • app servers scale (almost) linearly • use same hardware (if possible) • one type of processor, memory, disk • simplifies maintenance / repair
  39. 39. Bottleneck: application • lots of concurrent users • rails is single threaded (mutex lock) • solution: add more mongrels • rule of thumb: 10 mongrels per cpu • twitter runs on ca. 300 mongrels
  40. 40. Bottleneck: database • avoid to hit database • cache as much as possible • scale up database masters • more disks = faster access • add read-only db slaves
  41. 41. Scaling 2.0
  42. 42. Scaling storage • disk failure very common • use RAID1 for database masters • network storage • example: ATA-over-Ethernet (AoE) • accessible as block device • combine with cluster file system
  43. 43. Coraid EtherDrive, 11 TB, 200MB/s
  44. 44. • Evolving a cluster setup • Scaling • Make or buy? • Discussion
  45. 45. Make or buy? • think about total cost of ownership (TCO) • not only hardware costs • rackspace, power, bandwidth • administration overhead • „Utility? Or generator in the back yard?“ (David Young)
  46. 46. Our cluster • hardware: 3340 € • colocation with burst.net (for 1 year) • 60 $ + 12 x 140 $ = 900 $ = 670 € • 4000 € + administration
  47. 47. Rails hosting • example: Engine Yard • 1 slice = 1 CPU core, 400 MB ram • 4 slices (for 1 year) • 796 $ + 12 x 1116 $ = 14.200 $ • 10.500 €
  48. 48. • Evolving a cluster setup • Scaling • Make or buy? • Discussion
  49. 49. Summary • 2-machine cluster is cool (and quite affordable) setup • virtualization is a Good Thing • scaling the database can be tricky • make or buy depends on concrete situation
  50. 50. Questions?
  51. 51. http://tim.lossen.de ❦ photo credits http://www.flickr.com/photos/shivayanamahohm/192322513 http://www.flickr.com/photos/nicholas_t/286352669 http://www.flickr.com/photos/redjar/480709870

×