This is a talk given by Jason Hoffman at a workshop given by Joyent called "Scale With Rails" in 2006. There's quite a bit of prescience in this presentation, including the first documented use of ZFS in production ("Fsck you if you think ZFS isn't production") and of OS-based virtualization (zones) in the cloud (which, it must be said, was not called "cloud" in 2006).
2. The Guys {Thanks}
David Young, Dean Allen, Jason Hoffman, Matt
Imbach, Michael Koziarski, Scott Barron, Chris
Morris, Luke Crawford, Justin French, Johan
Sörensen, Ben Myles, Marten Veldthuis, Bryan
Bell, Ryan Schwartz, Jan Isley, Florian Munz,
Filip Hajný, Daniel Crowell, Christopher Horrell,
Terrell Russell, Peter Watridge, Josh Roebuck
3. Today
• Please humor me => a 5 minute story with too
many slides about how I came to stand here in
front of you
• Somethings about the Ruby on Rails site
• An accounting what’s buzzing in my ears all the
time
• Some examples with numbers
• How to think about “scaling”
• To talk about where we’ve taken our
infrastructure
9. So we thought let’s involve some
friends
• Dean Allen (Textpattern)
• Matt Mullenweg (Wordpress)
• Brad Choate (doing MT hosting + plugins)
• David Heinemeier Hansson (Instiki and “Rails”)
• Rickard of PunBB
• Allan => Textmate
19. But sometimes it’s gotten “bad”
• Ruby on Rails
• Turbogears
• Textmate
• All have videos (”screencasts”)
• They’ve been in the same paragraph on
Slashdot
• Then we’ve pushed 150-200 Mbps
• Open Source is free?
21. OK, So what?
• We found ourselves needing an infrastructure
for “diverse” things
• But fundamentally we’re talking about web
stuff, mail stuff, database stuff, storing and
moving around files stuff
22. And why are we here?
• One indication: because I get, on average, 2
emails/day asking the same questions
• That’s just “me”
• Support system and “sales” gets them as well
23. Examples from last week
• I have yet to find any examples of websites that have heavy traffic and
stream media that run from a Ruby on Rails platform, can you suggest
any sites that will demonstrate that the ruby platform is stable and
reliable enough to use on a commercial level?
• We are concerned about the long-term viability of Ruby on Rails as a
development language/environment.
• How easily can a ruby site be converted to another language? (If for any
reason we were forced to abandon ruby at some point in the future or I
can’t find someone to work with our code?).
• My company has some concerns on whether or not Ruby on Rails is the
right platform to deploy on if we have a very large scale app.
I could go on for a while.
24. So what is?
• A “large scale” application?
• Do many of us really have one? I mean a BIG
ONE?
• What is an “intensive” application?
• What is an “enterprise” application?
25. But we’ve been doing “it” with
• Perl
• PHP
• Python
• Java
• Does the language really really matter to the
system’s folks?
26. Ah-ha!
• System’s Folks you say?
• What are these “system administrators” that you
speak of?
• Does the network really matter?
• You mean there’s “networking people”?
• You mean it takes more than just having my
designer(s) learns The Rails?
27. If I had a nickel for every time ...
• “We’re going to need to scale this up to Flickr-
sized proportions!”
• “This could go very very fast!”
• “The market is HUGE”
28. Reality Check
• “OK”, I say
• “Not a problem”, I say
• “I can do that”, I say
• “For what you’re asking, I’ll need $325,000
tomorrow to start, it’ll take $18,500 a month to
run and you can expect that to go up along with
your growth.”
• “It’s a good rule of thumb to try and keep it
about 10% of your revenue once you’re going
because people ended being the most
expensive”
29. Oh. But we don’t have that kind
of money. And the app is free to
start.
31. What a minute!
• Why are you going on and on about this not
worry about scaling til you need to scale stuff?
• I can read this same thing on what’s their face’s
weblog.
• It’s not what I’m asking about!
• I need to know if I can “scale” with ruby!
• We’re not a start-up, why are you talking about
this from a “start-up’s” perspective?!
• I need a ...
32. A What?
• A framework for thinking about the entire
system?
• You mean it’s more than just the language?
• It’s more than about a readily available IDE?
• You mean one’s “development” framework is
just one small part?
• My system’s guys can scale anything.
34. And oh yes ... you should worry
still worry about scaling
35. Because what does “scaling” actually
mean?
• For a little software application company?
• Can you do the Start-up => Mid-sized, “small
caps” company transition without going out of
business?
• In a big company?
• Can you do what you gotta do without having
your program cut?
• Will your app run on a phone? Scale down.
38. The slowest part ?
• The fast ethernet or gigabit network port
(assuming there’s more than one drive)
• Transactions of something/second
39. • 1 Gbps = 125 MB/sec (100 Mbps = 12.5 MB/sec)
• The question is can your OS and our CPUs
push it?
• And let’s say you can, just how much is a Gbps
in some kind of other thing?
40. We used to use FreeBSD
Moral of the story? Even restricted to a single processor with a
single core, Solaris Nevada Build 31 can push 60% of a gigabit
connection. FreeBSD can’t.
41. OK, Solaris Good.
What’s a 100 Mbps in normal web traffic?
• Textdrive.com => 122 KB and has 20 objects
(~125KB for an uncached page view).
• 125KB page => 100-1000 unique visitors per
second
• 20 objects per page, that is 2000 requests per
second that could pump out of that system.
• Maximum.
42. • What is the ability to do 2000 requests/second
then?
• ((2000 requests/sec)*(20 requests per page)) *
0.125MB per page = 12.5 MB/sec (100Mbps
constant).
• 86400 seconds/day on 100Mbps => 8,640,000
uniques in a day with 172,800,00 hits.
43. • $8000/month
• Give or take a couple of grand depending
How much does just a 100 Mbps
commit cost?
44. Can you do 2000 requests/second?
• Sure an Apache, Lighttpd or Litespeed can do
1000-15,000 static or proxy requests/second
45. How do you do 2000 requests/
second?
• Alistapart.com bursts to that with Zeldman’s
“Web 3.0” for over an hour
• 10 Lighttpds => 200 proxy requests/second
each
• 1 Lighttpd => 40 request/sec x 5 Rails-FCGI
each
• We cached everything but writing to MySQL for
comments and that kept it on a 3.2 Ghz P4 with
4GB of RAM
46. The Shared Hosting
• Is this odd heterogenous largely ruby “application”
• Users received 8,641,866 emails, they sent 5,860,769 emails and had
14,984,680 pieces of spam blocked. Mail through the system averages
300 emails/minute with bursts up to 60,000 emails/minute.
• Websites cumulatively did ~400,000,000 page views.
47.
48. What is it running now?
• There’s 22 TBs of strongspace
• Fiber-attached EMC storage
• Migrated to Solaris “11”
• One big ZFS pool with LUNs
• Migrated down to one “hot” server: Dual 3.0
Ghz Xeon with 8GB of RAM
• Apache, Lighttpd, static FCGI
51. “Logical” servers for the hosted
connector
1) Jumpstart/Boot & Administrative servers (x2 per setup)
2) DHCP/LDAP for server identification/authentication and control (x2)
3) DNS: DNSCache for outside resolution (DNSCache as resolver) and a private DNS system
4) DNS MySQL (x2, master/slave, innodb tables)
5) SPAM filtering servers (running DSPAM -> files to NFS store and tracking to postgresql) (x4)
6) SPAM database setup (running postgresql) (x2)
7) SPAM NFS store (x2 heads clustered)
8) SMTP gateway out (x2)
9) SMTP gateway in (x4 -> delivery to Maildir over NFS)
10) Mail NFS store (x2 heads cluster) *main user file store
11) IMAP proxy servers (x2)
12) IMAP servers (x6)
13) User LDAP servers (x2 with hitting postgresql DB backend)
14) User LDAP-postgresql db (x2)
15) User postgresql DB servers (x4)
16) User Web/Application servers (running with kernel SSL accelerators) (x6)
17) User File Storage (NFS dual cluster heads) (x2)
18) Joyent Organization Provisioning server
62. Back to simple
Console server
One kind of switch
One kind of server
One kind of storage
One kind of interconnect
One kind of power plug
One kind of power strip
72. DIY: How do you start?
20amp 208V AC power
L6-20R
Street price: ~$50,000 co-lo’ed
73. Why DIY? Or at least a visit?
Outsourcing!You can whois yourself?
74. Some final things
• Keep it simple even if you’re starting out big
• Keep it simple even if you’re starting out small
• Our “stacks” are currently:
• Solaris, Mongrel, Apache 2.2 event, kssl
• Solaris, FCGI, Apache 2.2 event, kssl
• Solaris, FCGI, Lighttpd, kssl
• Solaris, FCGI, Lighttpd, Big-IP
• Zones, zones, zones