Hosting and the Woes

       by Jamie van Dyke
  Engine Yard Developer (UK)
Jamie van Dyke


My name is Jamie van Dyke, and in case you can see me from the back, I look ...
like this fella here. I used to be the ...
support


...Application Support Manager for Europe, but now I’ve transitioned over to ...
develop cool stuff


the Automation Development team over in ...
England


...the UK. ( So we do 24 hour development as well as support now ;-)
...just in case you forgot about us, we’re this little island here...we don’t have as many nukes
as you...but we’ve got Go...
Tom Mornini


...Tom Mornini, who looks like...
...this. Study the face well and feel free to grab him at any time during the conference to talk
shop, or just question ab...
The Chief


...CTO, Tom has experience in...
jack


every technology at Engine Yard...and can put many to shame...
of all trades


...in most of these areas. Moving on, we have...
Taylor Weibley


...Taylor Weibley. Taylor looks...
...like this. Taylor is our...
king


...Application Support Director. He’s the man responsible for keeping everyone happy, so if
you’re a customer and y...
Rails ++


...areas, of course Rails is one of them, but we consider him to be...
...one of our Supermen in the App Support Team. Finally, we have...
Edward Muller


...Edward Muller. Edward looks like...
...this, and he’s one of those incredibly clever people you meet that...
fluent in binary


...is able to talk to a computer instead of typing on the keyboard. Ed was and to some
degree still is o...
cluster ninja


...Cluster Engineer. After Jayson Vantuyl conceived and built ey00, our first cluster, Edward
went on to bu...
automation manager


... Manager, in the automation department. We basically improve the lives of the Engine Yard
sta, by ...
/intro


Enough about us, let’s talk about some of the problems that our customers have bumped
into and how to solve them.
Problems


Through hundreds of customers and hours of debugging, we’ve found a recurring set of
problems that crop up with...
active_record


...is Active Record. Too many times...
...I’ve seen a find(:all) that goes crazy and chews up memory like there’s unlimited banks of
it. You all know the solution...
development


...on a development machine you have a small set...
20 records


...of records. Your tests run against them and everything works fine. Your machine has 2Gb
of RAM and you don’...
production


...in production your site might be really popular and you’re running...
millions of records


...millions of users and assets for each of those. Running a find(:all) on that would probably
cause ...
indexes


...properly indexed your tables. At Engine Yard we use a team of DBA’s and the MySQL
Performance Blog Experts to...
plugins


There are a handful of technologies and plugins that cause problems for a lot of our
customers too. For example...
ferret


...the popular ferret indexing engine along with numerous dierent plugins that provide
interfaces to it, is const...
sphinx


...switch over to sphinx, which has a lot less problems (if any) and does the same job. We’ve
seen ferret indexes...
image science


image_science is great but there are a couple of gotchas for when you shift to using it in
production, wit...
hodel 3000


...hodel compliant logging allows you to use a specific log format that reporting tools can
handle. However, o...
...ever wonder what kind of trafic you can expect from dierent media coverages? okay, let’s
have a look at a few...
can haz trafficz?


...ever wonder what kind of trafic you can expect from dierent media coverages? okay, let’s
have a look...
Digg


so you get featured on digg’s homepage, how much trafic? Well, you’re looking...
10,000+ visitors


...at tens of thousands of visitors...but they’re all just checking out what’s new and featured,
becaus...
signups --


...you’ll rarely get many signups to your service. It’s all about seeing what’s new. Which is
exactly what we...
TechCrunch


...about TechCrunch. The only dierence being the level of trafic, which means more likely...
1000+ visitors


...thousands of visitors, you might hit tens of thousands but it depends on what your site is
all about. ...
Ruby Inside/Flow


...like Ruby Inside or Ruby Flow, will send you between...
500-1000 visitors


...five hundred and a thousand visitors a day, and you’ll get that for a few days to a week,
you’ll als...
signups --


...no signups from that. Which if you’re a ruby/rails specific site, will be higher than if you’re
not. It’s a...
The Today Show


...the Today Show, which is a great trafic provider, if you’re hosted with a provider who can
scale you q...
100,000+ visitors


...a huge number of visitors which is dependent on your site content, of course. One of our
customers ...
10,000 signups


...them ten thousand signups in the first hour alone. So this is definitely one of the media
spots you want...
Fox News
                 Business Show

...that could do you well, is Fox News Business Show. From just showing a site ur...
2000+ connections


...a whole chunk of visitors. 2000-3000 simultaneous visitors come rushing your way. Pray
to the Gods ...
$10,000,000


The ten million dollar question is always, how many users can one of our slices handle. Well,
how long is a ...
can you code?


Of course it’s all dependent on what your code quality is like. We can say that an average
Rails applicati...
queries per page


So, take a look at how many queries are running per page...
cache cache cache


... cache wherever possible...
file io


...keep file io down to a minimum...
separate services


...and if you can, move your services on to dierent servers. Like assets in one place,
application in ...
goodies


Let’s wrap up the talk with a mention of some open source goodies we have that you can
download...then we’ll mov...
eycap
                   (gems.engineyard.com)




This is a gem we developed that extends capistrano, it allows logging o...
gem source -a http://gems.engineyard.com
           gem install eycap --no-rdoc --no-ri
              (in deploy.rb) requi...
...these.
ask us anything


We filled the rest of the presentation with a q  a section, which went extremely well. Thank
you to every...
Rails Hosting and the Woes (Engine Yard)
Upcoming SlideShare
Loading in …5
×

Rails Hosting and the Woes (Engine Yard)

5,018 views

Published on

Common problems customers have and solutions, along with examples of traffic sizes from different media coverage.

Published in: Technology
2 Comments
2 Likes
Statistics
Notes
  • Find coupons for your hosting. Get a promo deal before you purchase hosting http://www.scriptcoupons.com/Vps.net/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I've added notes to the slides, so you get a rough idea of what was said during the presentation.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
5,018
On SlideShare
0
From Embeds
0
Number of Embeds
99
Actions
Shares
0
Downloads
22
Comments
2
Likes
2
Embeds 0
No embeds

No notes for slide

Rails Hosting and the Woes (Engine Yard)

  1. Hosting and the Woes by Jamie van Dyke Engine Yard Developer (UK)
  2. Jamie van Dyke My name is Jamie van Dyke, and in case you can see me from the back, I look ...
  3. like this fella here. I used to be the ...
  4. support ...Application Support Manager for Europe, but now I’ve transitioned over to ...
  5. develop cool stuff the Automation Development team over in ...
  6. England ...the UK. ( So we do 24 hour development as well as support now ;-)
  7. ...just in case you forgot about us, we’re this little island here...we don’t have as many nukes as you...but we’ve got Gordon Ramsay who’s even scarier! Enough about me, here’s...
  8. Tom Mornini ...Tom Mornini, who looks like...
  9. ...this. Study the face well and feel free to grab him at any time during the conference to talk shop, or just question about anything. Tom is our...
  10. The Chief ...CTO, Tom has experience in...
  11. jack every technology at Engine Yard...and can put many to shame...
  12. of all trades ...in most of these areas. Moving on, we have...
  13. Taylor Weibley ...Taylor Weibley. Taylor looks...
  14. ...like this. Taylor is our...
  15. king ...Application Support Director. He’s the man responsible for keeping everyone happy, so if you’re a customer and you’re not (is that even possible?), he’s the man to speak to. He has tonnes of skills in many...
  16. Rails ++ ...areas, of course Rails is one of them, but we consider him to be...
  17. ...one of our Supermen in the App Support Team. Finally, we have...
  18. Edward Muller ...Edward Muller. Edward looks like...
  19. ...this, and he’s one of those incredibly clever people you meet that...
  20. fluent in binary ...is able to talk to a computer instead of typing on the keyboard. Ed was and to some degree still is our...
  21. cluster ninja ...Cluster Engineer. After Jayson Vantuyl conceived and built ey00, our first cluster, Edward went on to build all the others and make improvements along the way. He’s now my ...
  22. automation manager ... Manager, in the automation department. We basically improve the lives of the Engine Yard sta, by automating the processes they do every day. For example, deploys can take a few hours manually, but with a sprinkling of automation we can pull that down to a few minutes.
  23. /intro Enough about us, let’s talk about some of the problems that our customers have bumped into and how to solve them.
  24. Problems Through hundreds of customers and hours of debugging, we’ve found a recurring set of problems that crop up with Rails development, and we’d like to make you aware of them if you aren’t already. No surprise, top of the list...
  25. active_record ...is Active Record. Too many times...
  26. ...I’ve seen a find(:all) that goes crazy and chews up memory like there’s unlimited banks of it. You all know the solution to this, in views you use pagination and in processing loops you use batches. The main cause of this going unnoticed before hitting production, is that...
  27. development ...on a development machine you have a small set...
  28. 20 records ...of records. Your tests run against them and everything works fine. Your machine has 2Gb of RAM and you don’t sit watching the memory usage as you test. However...
  29. production ...in production your site might be really popular and you’re running...
  30. millions of records ...millions of users and assets for each of those. Running a find(:all) on that would probably cause a problem. :-/ So paginate, and make sure you have...
  31. indexes ...properly indexed your tables. At Engine Yard we use a team of DBA’s and the MySQL Performance Blog Experts to help our customers get the best speed out of their databases.
  32. plugins There are a handful of technologies and plugins that cause problems for a lot of our customers too. For example...
  33. ferret ...the popular ferret indexing engine along with numerous dierent plugins that provide interfaces to it, is constantly corrupting indexes. Initially you’d think that it’s multiple processes accessing the same index...but we always use ferret server, so it’s not that. Our solution is simple and eective...
  34. sphinx ...switch over to sphinx, which has a lot less problems (if any) and does the same job. We’ve seen ferret indexes at 5Gb that are reduced to 1Gb when moved to sphinx. In conjunction with Evan Weaver’s ultrasphinx plugin, this is the ideal solution.
  35. image science image_science is great but there are a couple of gotchas for when you shift to using it in production, with monit...Image Science requires environment variables (INLINE_DIR or HOME) set up to work, monit clears these variables...so either use a wrapper script or set them in your app manually.
  36. hodel 3000 ...hodel compliant logging allows you to use a specific log format that reporting tools can handle. However, out of the few plugins that are out there for it, they neglect to inform you that you *need* to set logging to :info, otherwise your logs will (default to debug), be absolutely huge, and they don’t need to be if you want your reporting tools to work.
  37. ...ever wonder what kind of trafic you can expect from dierent media coverages? okay, let’s have a look at a few...
  38. can haz trafficz? ...ever wonder what kind of trafic you can expect from dierent media coverages? okay, let’s have a look at a few...
  39. Digg so you get featured on digg’s homepage, how much trafic? Well, you’re looking...
  40. 10,000+ visitors ...at tens of thousands of visitors...but they’re all just checking out what’s new and featured, because...
  41. signups -- ...you’ll rarely get many signups to your service. It’s all about seeing what’s new. Which is exactly what we can say...
  42. TechCrunch ...about TechCrunch. The only dierence being the level of trafic, which means more likely...
  43. 1000+ visitors ...thousands of visitors, you might hit tens of thousands but it depends on what your site is all about. A Ruby specific site...
  44. Ruby Inside/Flow ...like Ruby Inside or Ruby Flow, will send you between...
  45. 500-1000 visitors ...five hundred and a thousand visitors a day, and you’ll get that for a few days to a week, you’ll also get virtually...
  46. signups -- ...no signups from that. Which if you’re a ruby/rails specific site, will be higher than if you’re not. It’s all whether your site is relevant to the market with these smaller sites, when I say smaller I mean not on the scale of ...
  47. The Today Show ...the Today Show, which is a great trafic provider, if you’re hosted with a provider who can scale you quickly and provide you with a team of Rails experts that will help you through it, by both scaling up your service, as well as providing quick advice on how to increase the eficiency of your pages. Someone like us ;-) You’re looking at...
  48. 100,000+ visitors ...a huge number of visitors which is dependent on your site content, of course. One of our customers called Catalog Choice got featured on the Today show and the trafic spike was enormous, as well as obtaining...
  49. 10,000 signups ...them ten thousand signups in the first hour alone. So this is definitely one of the media spots you want to get featured on. Another, not surprisingly, high trafic provider ...
  50. Fox News Business Show ...that could do you well, is Fox News Business Show. From just showing a site url on the tv, you can expect...
  51. 2000+ connections ...a whole chunk of visitors. 2000-3000 simultaneous visitors come rushing your way. Pray to the Gods that you’re ready to scale up for this one. ;-)
  52. $10,000,000 The ten million dollar question is always, how many users can one of our slices handle. Well, how long is a piece of string?
  53. can you code? Of course it’s all dependent on what your code quality is like. We can say that an average Rails application running over 3 mongrels, can trivially handle millions of unique visitors a month.
  54. queries per page So, take a look at how many queries are running per page...
  55. cache cache cache ... cache wherever possible...
  56. file io ...keep file io down to a minimum...
  57. separate services ...and if you can, move your services on to dierent servers. Like assets in one place, application in another, database in another, etc.
  58. goodies Let’s wrap up the talk with a mention of some open source goodies we have that you can download...then we’ll move on to the Q and A section.
  59. eycap (gems.engineyard.com) This is a gem we developed that extends capistrano, it allows logging of deploys, has tons of tasks for sphinx / ferret / memcached / etc. If you use nginx, or monit, or sphinx or....whatever, then this gem will be handy for you, download it using ...
  60. gem source -a http://gems.engineyard.com gem install eycap --no-rdoc --no-ri (in deploy.rb) require ‘eycap’ ...these commands, with sudo if you’re on a mac...and you get lots of capistrano goodies...like
  61. ...these.
  62. ask us anything We filled the rest of the presentation with a q a section, which went extremely well. Thank you to everyone who came and participated.

×