Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Joyent circa 2006 (Scale with Rails)

3,611 views

Published on

This is a talk given by Jason Hoffman at a workshop given by Joyent called "Scale With Rails" in 2006. There's quite a bit of prescience in this presentation, including the first documented use of ZFS in production ("Fsck you if you think ZFS isn't production") and of OS-based virtualization (zones) in the cloud (which, it must be said, was not called "cloud" in 2006).

Published in: Technology
  • Be the first to comment

Joyent circa 2006 (Scale with Rails)

  1. 1. Scale with Rails Real-life capacity and deployment planning for small to large scale Rails applications
  2. 2. The Guys {Thanks} David Young, Dean Allen, Jason Hoffman, Matt Imbach, Michael Koziarski, Scott Barron, Chris Morris, Luke Crawford, Justin French, Johan Sörensen, Ben Myles, Marten Veldthuis, Bryan Bell, Ryan Schwartz, Jan Isley, Florian Munz, Filip Hajný, Daniel Crowell, Christopher Horrell, Terrell Russell, Peter Watridge, Josh Roebuck
  3. 3. Today • Please humor me => a 5 minute story with too many slides about how I came to stand here in front of you • Somethings about the Ruby on Rails site • An accounting what’s buzzing in my ears all the time • Some examples with numbers • How to think about “scaling” • To talk about where we’ve taken our infrastructure
  4. 4. Who?
  5. 5. Internet beginnings for me
  6. 6. I liked collaboration
  7. 7. It started with Textpattern The best CMS/blog thing written in PHP ever
  8. 8. We used basecamp Ooooooh, it’s in Ruby? Reeeeeeeeeally.
  9. 9. So we thought let’s involve some friends • Dean Allen (Textpattern) • Matt Mullenweg (Wordpress) • Brad Choate (doing MT hosting + plugins) • David Heinemeier Hansson (Instiki and “Rails”) • Rickard of PunBB • Allan => Textmate
  10. 10. I was a committer once
  11. 11. What did they get me into? 5/2003 9/2005 5/2004 “it” started
  12. 12. Books are important Not going to pay in my house. Send those books out.
  13. 13. Rails
  14. 14. rubyonrails.com ~3500-4500 TBs/month (~10-15Mbps)
  15. 15. But sometimes it’s gotten “bad” • Ruby on Rails • Turbogears • Textmate • All have videos (”screencasts”) • They’ve been in the same paragraph on Slashdot • Then we’ve pushed 150-200 Mbps • Open Source is free?
  16. 16. Some of Joyent’s We combined last year to be the “perfect” company.
  17. 17. OK, So what? • We found ourselves needing an infrastructure for “diverse” things • But fundamentally we’re talking about web stuff, mail stuff, database stuff, storing and moving around files stuff
  18. 18. And why are we here? • One indication: because I get, on average, 2 emails/day asking the same questions • That’s just “me” • Support system and “sales” gets them as well
  19. 19. Examples from last week • I have yet to find any examples of websites that have heavy traffic and stream media that run from a Ruby on Rails platform, can you suggest any sites that will demonstrate that the ruby platform is stable and reliable enough to use on a commercial level? • We are concerned about the long-term viability of Ruby on Rails as a development language/environment. • How easily can a ruby site be converted to another language? (If for any reason we were forced to abandon ruby at some point in the future or I can’t find someone to work with our code?). • My company has some concerns on whether or not Ruby on Rails is the right platform to deploy on if we have a very large scale app. I could go on for a while.
  20. 20. So what is? • A “large scale” application? • Do many of us really have one? I mean a BIG ONE? • What is an “intensive” application? • What is an “enterprise” application?
  21. 21. But we’ve been doing “it” with • Perl • PHP • Python • Java • Does the language really really matter to the system’s folks?
  22. 22. Ah-ha! • System’s Folks you say? • What are these “system administrators” that you speak of? • Does the network really matter? • You mean there’s “networking people”? • You mean it takes more than just having my designer(s) learns The Rails?
  23. 23. If I had a nickel for every time ... • “We’re going to need to scale this up to Flickr- sized proportions!” • “This could go very very fast!” • “The market is HUGE”
  24. 24. Reality Check • “OK”, I say • “Not a problem”, I say • “I can do that”, I say • “For what you’re asking, I’ll need $325,000 tomorrow to start, it’ll take $18,500 a month to run and you can expect that to go up along with your growth.” • “It’s a good rule of thumb to try and keep it about 10% of your revenue once you’re going because people ended being the most expensive”
  25. 25. Oh. But we don’t have that kind of money. And the app is free to start.
  26. 26. OK.
  27. 27. What a minute! • Why are you going on and on about this not worry about scaling til you need to scale stuff? • I can read this same thing on what’s their face’s weblog. • It’s not what I’m asking about! • I need to know if I can “scale” with ruby! • We’re not a start-up, why are you talking about this from a “start-up’s” perspective?! • I need a ...
  28. 28. A What? • A framework for thinking about the entire system? • You mean it’s more than just the language? • It’s more than about a readily available IDE? • You mean one’s “development” framework is just one small part? • My system’s guys can scale anything.
  29. 29. Yes It Is More
  30. 30. And oh yes ... you should worry still worry about scaling
  31. 31. Because what does “scaling” actually mean? • For a little software application company? • Can you do the Start-up => Mid-sized, “small caps” company transition without going out of business? • In a big company? • Can you do what you gotta do without having your program cut? • Will your app run on a phone? Scale down.
  32. 32. Let’s talk some details
  33. 33. First, let’s talk about limiting factors
  34. 34. The slowest part ? • The fast ethernet or gigabit network port (assuming there’s more than one drive) • Transactions of something/second
  35. 35. • 1 Gbps = 125 MB/sec (100 Mbps = 12.5 MB/sec) • The question is can your OS and our CPUs push it? • And let’s say you can, just how much is a Gbps in some kind of other thing?
  36. 36. We used to use FreeBSD Moral of the story? Even restricted to a single processor with a single core, Solaris Nevada Build 31 can push 60% of a gigabit connection. FreeBSD can’t.
  37. 37. OK, Solaris Good. What’s a 100 Mbps in normal web traffic? • Textdrive.com => 122 KB and has 20 objects (~125KB for an uncached page view). • 125KB page => 100-1000 unique visitors per second • 20 objects per page, that is 2000 requests per second that could pump out of that system. • Maximum.
  38. 38. • What is the ability to do 2000 requests/second then? • ((2000 requests/sec)*(20 requests per page)) * 0.125MB per page = 12.5 MB/sec (100Mbps constant). • 86400 seconds/day on 100Mbps => 8,640,000 uniques in a day with 172,800,00 hits.
  39. 39. • $8000/month • Give or take a couple of grand depending How much does just a 100 Mbps commit cost?
  40. 40. Can you do 2000 requests/second? • Sure an Apache, Lighttpd or Litespeed can do 1000-15,000 static or proxy requests/second
  41. 41. How do you do 2000 requests/ second? • Alistapart.com bursts to that with Zeldman’s “Web 3.0” for over an hour • 10 Lighttpds => 200 proxy requests/second each • 1 Lighttpd => 40 request/sec x 5 Rails-FCGI each • We cached everything but writing to MySQL for comments and that kept it on a 3.2 Ghz P4 with 4GB of RAM
  42. 42. The Shared Hosting • Is this odd heterogenous largely ruby “application” • Users received 8,641,866 emails, they sent 5,860,769 emails and had 14,984,680 pieces of spam blocked. Mail through the system averages 300 emails/minute with bursts up to 60,000 emails/minute. • Websites cumulatively did ~400,000,000 page views.
  43. 43. What is it running now? • There’s 22 TBs of strongspace • Fiber-attached EMC storage • Migrated to Solaris “11” • One big ZFS pool with LUNs • Migrated down to one “hot” server: Dual 3.0 Ghz Xeon with 8GB of RAM • Apache, Lighttpd, static FCGI
  44. 44. Fsck You {if you think ZFS isn’t production}
  45. 45. Is it a web app or what? ~20 Mbps constant
  46. 46. “Logical” servers for the hosted connector 1) Jumpstart/Boot & Administrative servers (x2 per setup) 2) DHCP/LDAP for server identification/authentication and control (x2) 3) DNS: DNSCache for outside resolution (DNSCache as resolver) and a private DNS system 4) DNS MySQL (x2, master/slave, innodb tables) 5) SPAM filtering servers (running DSPAM -> files to NFS store and tracking to postgresql) (x4) 6) SPAM database setup (running postgresql) (x2) 7) SPAM NFS store (x2 heads clustered) 8) SMTP gateway out (x2) 9) SMTP gateway in (x4 -> delivery to Maildir over NFS) 10) Mail NFS store (x2 heads cluster) *main user file store 11) IMAP proxy servers (x2) 12) IMAP servers (x6) 13) User LDAP servers (x2 with hitting postgresql DB backend) 14) User LDAP-postgresql db (x2) 15) User postgresql DB servers (x4) 16) User Web/Application servers (running with kernel SSL accelerators) (x6) 17) User File Storage (NFS dual cluster heads) (x2) 18) Joyent Organization Provisioning server
  47. 47. So how do you scale that “web” application?
  48. 48. You know
  49. 49. That beautiful Connector application where only 2 of the 18 “things” is “rails”.
  50. 50. So most scale is how you scale those “common” services
  51. 51. And why just 3.0 Ghz?
  52. 52. CPUs are largely idle & RAM RAM RAM 08:23:01 PM CPU %user %nice %sys %iowait %irq %soft %idle intr/s 08:23:01 PM all 2.64 0.00 0.36 2.81 0.02 0.14 94.02 1239.49 08:23:01 PM 0 6.86 0.00 1.18 4.72 0.10 0.69 86.45 68.94 08:23:01 PM 1 2.63 0.00 0.34 1.95 0.01 0.11 95.01 192.97 08:23:01 PM 2 2.78 0.00 0.43 7.41 0.02 0.11 89.29 197.21 08:23:01 PM 3 2.16 0.00 0.29 4.12 0.01 0.07 93.39 178.35 08:23:01 PM 4 1.67 0.00 0.17 1.09 0.01 0.04 97.04 117.29 08:23:01 PM 5 1.67 0.00 0.17 1.11 0.01 0.04 97.05 161.00 08:23:01 PM 6 1.67 0.00 0.16 1.04 0.01 0.04 97.13 161.86 08:23:01 PM 7 1.68 0.00 0.16 1.08 0.01 0.04 97.07 161.86 • “Busy” mysql database server • ~240GB of data with ~150,000 users • Swapping just a little little bit, so ...
  53. 53. So what kind of CPUs have we moved into?
  54. 54. MB/CMP0/P0 0 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P1 1 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P2 2 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P3 3 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P4 4 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P5 5 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P6 6 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P7 7 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P8 8 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P9 9 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P10 10 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P11 11 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P12 12 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P13 13 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P14 14 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P15 15 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P16 16 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P17 17 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P18 18 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P19 19 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P20 20 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P21 21 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P22 22 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P23 23 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P24 24 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P25 25 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P26 26 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P27 27 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P28 28 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P29 29 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P30 30 1000 MHz SUNW,UltraSPARC-T1 MB/CMP0/P31 31 1000 MHz SUNW,UltraSPARC-T1 Sun T1000’s Ultrasparc T1 32 logical 1.0 Ghz Combined with many software instances Zones Best match between actual CPU and RAM usage
  55. 55. It’s not really about high frequencies If you buy intel, buy weak ones, spend the money on RAM Throughput!
  56. 56. Not simple
  57. 57. Back to simple Console server One kind of switch One kind of server One kind of storage One kind of interconnect One kind of power plug One kind of power strip
  58. 58. Well. We do have some load balancers still.
  59. 59. There’s actually lots of different kinds of plugs
  60. 60. What the “default” 5-15R gets you Umm ... ok so there’s a real disconnect between server power and datacenter “design”
  61. 61. I/O Storage PublicRemote Interconnects Jumbo Frames Own Switches
  62. 62. 4 “old” San Diego Racks Per Rack: 39 processors 39 logical CPUs 52 GB RAM 5280 watts Total: 156 logical CPUs 208 GB RAM 21,120 watts
  63. 63. Yes that is what they look like
  64. 64. Become a new node 1152 logical 1Ghz CPUs 576 GB RAM 6,480 watts 31 amps @ 208V 2 x 20amp @ 208V L6-20R 2 x 24 plug 20amp/208V ---------------------------------- 12 Enclosures 36 controllers 180 drives 90 TBs raw storage 77 TBs clustered storage 29.4 amps @ 208V 2 x 20amp @ 208V L6-20R 2 x 24 plug 20amp/208V Per Rack: 39 processors 39 logical CPUs 52 GB RAM 5280 watts Total: 156 logical CPUs 208 GB RAM 21,120 watts
  65. 65. Server + Storage RAID5 7 TBs RAID5 7 TBs RAID5 7 TBs RAIDZ ZFS Pool iSCSI
  66. 66. This is going to be open for you to use
  67. 67. DIY: How do you start? 20amp 208V AC power L6-20R Street price: ~$50,000 co-lo’ed
  68. 68. Why DIY? Or at least a visit? Outsourcing!You can whois yourself?
  69. 69. Some final things • Keep it simple even if you’re starting out big • Keep it simple even if you’re starting out small • Our “stacks” are currently: • Solaris, Mongrel, Apache 2.2 event, kssl • Solaris, FCGI, Apache 2.2 event, kssl • Solaris, FCGI, Lighttpd, kssl • Solaris, FCGI, Lighttpd, Big-IP • Zones, zones, zones

×