presented at 'ostrava on rails' on 2007-06-22. there exists also a video recording: http://www.rails.cz/articles/2007/06/27/oor-videos

1. How to build a
state-of-the-art
rails cluster
Tim Lossen
Infopark AG, Berlin

2. You mean ...
like this?

3. System X
• Virginia Tech
• 1100 x Apple XServe G5
• 2 CPUs each (2,3 GHz)
• Infiniband network
• 12 Teraflops

4. How to build a
state-of-the-art
rails cluster
for 100.000 Koruny
Tim Lossen
Infopark AG, Berlin

5. 100.000 Koruny
=
4700 $
=
3500 €

6. • Evolving a cluster setup
• Scaling
• Make or buy?
• Discussion

7. Aims
• performance
• efficiency
• availability
• scalability

8. Performance
• pages / second
• throughput
• 10 Mbit connection = max. about 1 million
unique page impressions per day

9. Efficiency
• bang per buck
• bang per watt

10. Availability
• uptime
• can be expressed in ‚nines‘
• 99% (2 nines) = max. 100 minutes
downtime per week
• 99,9999% (6 nines) = max. 0,6 seconds
downtime per week

11. Scalability
• grow (or shrink) app as needed
• ideal: linear with number of machines

12. Setup 1

13. Example software
• nginx
• mongrel cluster
• mysql

14. Ideas
• put everything on one machine
• use top quality hardware
• go for high processor speed

15. Example hardware
• Sun Fire X2200 M2
• 2 x Opteron 2220 (dual core / 2,8 GHz)
• 4 GB ram
• 250 GB disk
• 3290 €

16. Result
• performance ✔
• efficiency
• availability
• scalability

17. Setup 2

18. Ideas
• put app and database on separate machines
• use cheap commodity hardware
• like Google!
• go for high number of CPUs

19. Example hardware
• Dell PowerEdge SC 1435
• 2 x Opteron 2212 (dual core / 2 GHz)
• 4 GB ram, 160 GB disk
• 2 machines = 8 cores
• 3340 €

20. Result
• performance ✔
• efficiency ✔
• availability
• scalability

21. Setup 3

22. Ideas
• cluster of identical nodes
• loadbalancing
• failover
• eliminate single points of failure through
redundancy

23. Setup 3a

24. Virtualization
• partition one physical machine into multiple
virtual machines
• assign dedicated ressources (cpu, ram) to
virtual machines
• example: xen (linux), zones (solaris)

25. Why virtualization?
• flexibility
• simplified deployment
• move images between machines
• optimization
• hardware utilization
• tuning

26. Why virtualization?
• isolation
• use different operating systems
• contain intrusion
• multiple environments (staging, production)
• but: small performance overhead (ca. 5%)

27. Setup 3a

28. Loadbalancer
• openBSD
• CARP: shared virtual ip, automatic failover
• pf: firewall + loadbalancer
• openBSD on xen: currently only on ramdisk
• better use dedicated hardware instead

29. Redundant firewall

30. Database
• mysql
• master-master replication
• caveat: avoid primary key clashes!
• other possibilities:
• master-slave replication
• mysql cluster

31. Result
• performance ✔
• efficiency ✔
• availability ✔
• scalability ?

32. • Evolving a cluster setup
• Scaling
• Make or buy?
• Discussion

33. Scaling
• not a rails-specific
problem
• find and eliminate
bottlenecks!

34. Scaling approaches
• scaling up
• increase power of machines
• scaling out
• increase number of machines

35. Scaling up: memory
• add more memory
• mongrel can be very memory-hungry
• useful for caching (memcached)
• hint: check crucial.com for low prices

36. Scaling up: CPU
• upgrade to quad-core Opterons
• available later this year
• same socket
• same power / heat envelope
• use as drop-in replacement

37. Scaling out

38. Scaling out
• app servers scale (almost) linearly
• use same hardware (if possible)
• one type of processor, memory, disk
• simplifies maintenance / repair

39. Bottleneck: application
• lots of concurrent users
• rails is single threaded (mutex lock)
• solution: add more mongrels
• rule of thumb: 10 mongrels per cpu
• twitter runs on ca. 300 mongrels

40. Bottleneck: database
• avoid to hit database
• cache as much as possible
• scale up database masters
• more disks = faster access
• add read-only db slaves

41. Scaling 2.0

42. Scaling storage
• disk failure very common
• use RAID1 for database masters
• network storage
• example: ATA-over-Ethernet (AoE)
• accessible as block device
• combine with cluster file system

43. Coraid EtherDrive, 11 TB, 200MB/s

44. • Evolving a cluster setup
• Scaling
• Make or buy?
• Discussion

45. Make or buy?
• think about total cost of ownership (TCO)
• not only hardware costs
• rackspace, power, bandwidth
• administration overhead
• „Utility? Or generator in the back
yard?“ (David Young)

46. Our cluster
• hardware: 3340 €
• colocation with burst.net (for 1 year)
• 60 $ + 12 x 140 $ = 900 $ = 670 €
• 4000 € + administration

47. Rails hosting
• example: Engine Yard
• 1 slice = 1 CPU core, 400 MB ram
• 4 slices (for 1 year)
• 796 $ + 12 x 1116 $ = 14.200 $
• 10.500 €

48. • Evolving a cluster setup
• Scaling
• Make or buy?
• Discussion

49. Summary
• 2-machine cluster is cool (and quite
affordable) setup
• virtualization is a Good Thing
• scaling the database can be tricky
• make or buy depends on concrete situation

50. Questions?

53. http://tim.lossen.de
❦
photo credits
http://www.flickr.com/photos/shivayanamahohm/192322513
http://www.flickr.com/photos/nicholas_t/286352669
http://www.flickr.com/photos/redjar/480709870