Hosting and the Woes
by Jamie van Dyke
Engine Yard Developer (UK)

Jamie van Dyke
My name is Jamie van Dyke, and in case you can see me from the back, I look ...

like this fella here. I used to be the ...

support
...Application Support Manager for Europe, but now I’ve transitioned over to ...

develop cool stuff
the Automation Development team over in ...

England
...the UK. ( So we do 24 hour development as well as support now ;-)

...just in case you forgot about us, we’re this little island here...we don’t have as many nukes
as you...but we’ve got Gordon Ramsay who’s even scarier! Enough about me, here’s...

Tom Mornini
...Tom Mornini, who looks like...

...this. Study the face well and feel free to grab him at any time during the conference to talk
shop, or just question about anything. Tom is our...

The Chief
...CTO, Tom has experience in...

jack
every technology at Engine Yard...and can put many to shame...

of all trades
...in most of these areas. Moving on, we have...

Taylor Weibley
...Taylor Weibley. Taylor looks...

...like this. Taylor is our...

king
...Application Support Director. He’s the man responsible for keeping everyone happy, so if
you’re a customer and you’re not (is that even possible?), he’s the man to speak to. He has
tonnes of skills in many...

Rails ++
...areas, of course Rails is one of them, but we consider him to be...

...one of our Supermen in the App Support Team. Finally, we have...

Edward Muller
...Edward Muller. Edward looks like...

...this, and he’s one of those incredibly clever people you meet that...

fluent in binary
...is able to talk to a computer instead of typing on the keyboard. Ed was and to some
degree still is our...

cluster ninja
...Cluster Engineer. After Jayson Vantuyl conceived and built ey00, our first cluster, Edward
went on to build all the others and make improvements along the way. He’s now my ...

automation manager
... Manager, in the automation department. We basically improve the lives of the Engine Yard
staff, by automating the processes they do every day. For example, deploys can take a few
hours manually, but with a sprinkling of automation we can pull that down to a few minutes.

</intro>
Enough about us, let’s talk about some of the problems that our customers have bumped
into and how to solve them.

Problems
Through hundreds of customers and hours of debugging, we’ve found a recurring set of
problems that crop up with Rails development, and we’d like to make you aware of them if
you aren’t already. No surprise, top of the list...

active_record
...is Active Record. Too many times...

...I’ve seen a find(:all) that goes crazy and chews up memory like there’s unlimited banks of
it. You all know the solution to this, in views you use pagination and in processing loops you
use batches. The main cause of this going unnoticed before hitting production, is that...

development
...on a development machine you have a small set...

20 records
...of records. Your tests run against them and everything works fine. Your machine has 2Gb
of RAM and you don’t sit watching the memory usage as you test. However...

production
...in production your site might be really popular and you’re running...

millions of records
...millions of users and assets for each of those. Running a find(:all) on that would probably
cause a problem. :-/ So paginate, and make sure you have...

indexes
...properly indexed your tables. At Engine Yard we use a team of DBA’s and the MySQL
Performance Blog Experts to help our customers get the best speed out of their databases.

plugins
There are a handful of technologies and plugins that cause problems for a lot of our
customers too. For example...

ferret
...the popular ferret indexing engine along with numerous different plugins that provide
interfaces to it, is constantly corrupting indexes. Initially you’d think that it’s multiple
processes accessing the same index...but we always use ferret server, so it’s not that. Our
solution is simple and effective...

sphinx
...switch over to sphinx, which has a lot less problems (if any) and does the same job. We’ve
seen ferret indexes at 5Gb that are reduced to 1Gb when moved to sphinx. In conjunction
with Evan Weaver’s ultrasphinx plugin, this is the ideal solution.

image science
image_science is great but there are a couple of gotchas for when you shift to using it in
production, with monit...Image Science requires environment variables (INLINE_DIR or HOME)
set up to work, monit clears these variables...so either use a wrapper script or set them in
your app manually.

hodel 3000
...hodel compliant logging allows you to use a specific log format that reporting tools can
handle. However, out of the few plugins that are out there for it, they neglect to inform you
that you *need* to set logging to :info, otherwise your logs will (default to debug), be
absolutely huge, and they don’t need to be if you want your reporting tools to work.

...ever wonder what kind of traffic you can expect from different media coverages? okay, let’s
have a look at a few...

can haz trafficz?
...ever wonder what kind of traffic you can expect from different media coverages? okay, let’s
have a look at a few...

Digg
so you get featured on digg’s homepage, how much traffic? Well, you’re looking...

10,000+ visitors
...at tens of thousands of visitors...but they’re all just checking out what’s new and featured,
because...

signups --
...you’ll rarely get many signups to your service. It’s all about seeing what’s new. Which is
exactly what we can say...

TechCrunch
...about TechCrunch. The only difference being the level of traffic, which means more likely...

1000+ visitors
...thousands of visitors, you might hit tens of thousands but it depends on what your site is
all about. A Ruby specific site...

Ruby Inside/Flow
...like Ruby Inside or Ruby Flow, will send you between...

500-1000 visitors
...five hundred and a thousand visitors a day, and you’ll get that for a few days to a week,
you’ll also get virtually...

signups --
...no signups from that. Which if you’re a ruby/rails specific site, will be higher than if you’re
not. It’s all whether your site is relevant to the market with these smaller sites, when I say
smaller I mean not on the scale of ...

The Today Show
...the Today Show, which is a great traffic provider, if you’re hosted with a provider who can
scale you quickly and provide you with a team of Rails experts that will help you through it,
by both scaling up your service, as well as providing quick advice on how to increase the
efficiency of your pages. Someone like us ;-) You’re looking at...

100,000+ visitors
...a huge number of visitors which is dependent on your site content, of course. One of our
customers called Catalog Choice got featured on the Today show and the traffic spike was
enormous, as well as obtaining...

10,000 signups
...them ten thousand signups in the first hour alone. So this is definitely one of the media
spots you want to get featured on. Another, not surprisingly, high traffic provider ...

Fox News
Business Show
...that could do you well, is Fox News Business Show. From just showing a site url on the tv,
you can expect...

2000+ connections
...a whole chunk of visitors. 2000-3000 simultaneous visitors come rushing your way. Pray
to the Gods that you’re ready to scale up for this one. ;-)

$10,000,000
The ten million dollar question is always, how many users can one of our slices handle. Well,
how long is a piece of string?

can you code?
Of course it’s all dependent on what your code quality is like. We can say that an average
Rails application running over 3 mongrels, can trivially handle millions of unique visitors a
month.

queries per page
So, take a look at how many queries are running per page...

cache cache cache
... cache wherever possible...

file io
...keep file io down to a minimum...

separate services
...and if you can, move your services on to different servers. Like assets in one place,
application in another, database in another, etc.

goodies
Let’s wrap up the talk with a mention of some open source goodies we have that you can
download...then we’ll move on to the Q and A section.

eycap
(gems.engineyard.com)
This is a gem we developed that extends capistrano, it allows logging of deploys, has tons of
tasks for sphinx / ferret / memcached / etc. If you use nginx, or monit, or sphinx
or....whatever, then this gem will be handy for you, download it using ...

gem source -a http://gems.engineyard.com
gem install eycap --no-rdoc --no-ri
(in deploy.rb) require ‘eycap’
...these commands, with sudo if you’re on a mac...and you get lots of capistrano
goodies...like

...these.

ask us anything
We filled the rest of the presentation with a q & a section, which went extremely well. Thank
you to everyone who came and participated.