3. 2015: The Aftermath A Preview of Things to Come
‣ Rails app in multiple datacenters
4. 2015: The Aftermath A Preview of Things to Come
‣ Rails app in multiple datacenters
‣ Huge MySQL boxes
5. 2015: The Aftermath A Preview of Things to Come
‣ Rails app in multiple datacenters
‣ Huge MySQL boxes
‣ Over 1,000 controllers
6. 2015: The Aftermath A Preview of Things to Come
‣ Rails app in multiple datacenters
‣ Huge MySQL boxes
‣ Over 1,000 controllers
‣ Spotty test coverage
7. 2015: The Aftermath A Preview of Things to Come
‣ Rails app in multiple datacenters
‣ Huge MySQL boxes
‣ Over 1,000 controllers
‣ Spotty test coverage
‣ Sad developers
8. 2015: The Aftermath A Preview of Things to Come
‣ Rails app in multiple datacenters
‣ Huge MySQL boxes
‣ Over 1,000 controllers
‣ Spotty test coverage
‣ Sad developers
‣ Ultra-slow test suite
16. Actual Scaling Problems
‣ Scaling your data
‣ Scaling your codebase
‣ Scaling your customers
‣ Scaling your feature count
17. Actual Scaling Problems
‣ Scaling your data
‣ Scaling your codebase
‣ Scaling your customers
‣ Scaling your feature count
‣ Scaling your developer headcount
19. Blog in 15 Minutes!
1 app == 1 product == 1 developer
20. Blog in 15 Minutes!
1 app == 1 product == 1 developer
“Welcome aboard,
put your code here.”
21. Blog in 15 Minutes!
‣ Admin interface: /admin
‣ Analytics section: /trends
22. Blog in 15 Minutes!
‣ Admin interface: /admin
‣ Analytics section: /trends
“Rails can do anything!”
“Let’s make it do everything!”
23. Blog in 15 Minutes!
class ApplicationMailer < ActionMailer::Base
# 4 million redundant lines
end
24. Blog in 15 Minutes!
class User < ActiveRecord::Base
validates_presence_of :email
validates_presence_of :phone
validates_presence_of :bajillion_other_things
end
25. Blog in 15 Minutes!
“Nice mini-framework.
Throw that in ./lib !”
26. Blog in 15 Minutes!
“Nice mini-framework.
Throw that in ./lib !”
“Don’t forget tests
in ./spec/lib !”
42. Conway’s Law
"organizations which design systems ... are constrained to
produce designs which are copies of the communication
structures of these organizations"
{Intro}\nDid anybody go to the Square party? Yeah.\nI feel very lucky to work at Square. I feel surrounded by kind geniuses. If that sounds interesting to you we need people who can work with hardware, Android, iOS, Ruby, Java, Networking, Machine Learning, and anything else computer-related. Please come interview with us - we make the interview fun.\n\n
Okay, assuming you&#x2019;ve got a Rails app: let&#x2019;s fast forward 3 years from now.\nYou&#x2019;ve done well, you&#x2019;ve got lots of customers and you&#x2019;re up to 20 developers working full-time. Awesome. You&#x2019;re set.\n[*]\nYou now maintain a Rails application running in two datacenters [*] backed by one giant $15K MySQL box in each. [*] It's got 1,000 controllers and models. [*] It's covered by tests but you're not sure how well covered because running RCov would take 7 years. Or SEGFAULT. [*] Your 22 developers complain constantly that it's the worst thing about their job, that they aren't sure they like "working at big companies" and they're threatening to leave. [*] Your tests take so long that deploying even small fixes is hazardous and slow. Everybody wants to work on something new and fancy but they're stuck debugging "The Rails App."\n\n\n\n\n
Okay, assuming you&#x2019;ve got a Rails app: let&#x2019;s fast forward 3 years from now.\nYou&#x2019;ve done well, you&#x2019;ve got lots of customers and you&#x2019;re up to 20 developers working full-time. Awesome. You&#x2019;re set.\n[*]\nYou now maintain a Rails application running in two datacenters [*] backed by one giant $15K MySQL box in each. [*] It's got 1,000 controllers and models. [*] It's covered by tests but you're not sure how well covered because running RCov would take 7 years. Or SEGFAULT. [*] Your 22 developers complain constantly that it's the worst thing about their job, that they aren't sure they like "working at big companies" and they're threatening to leave. [*] Your tests take so long that deploying even small fixes is hazardous and slow. Everybody wants to work on something new and fancy but they're stuck debugging "The Rails App."\n\n\n\n\n
Okay, assuming you&#x2019;ve got a Rails app: let&#x2019;s fast forward 3 years from now.\nYou&#x2019;ve done well, you&#x2019;ve got lots of customers and you&#x2019;re up to 20 developers working full-time. Awesome. You&#x2019;re set.\n[*]\nYou now maintain a Rails application running in two datacenters [*] backed by one giant $15K MySQL box in each. [*] It's got 1,000 controllers and models. [*] It's covered by tests but you're not sure how well covered because running RCov would take 7 years. Or SEGFAULT. [*] Your 22 developers complain constantly that it's the worst thing about their job, that they aren't sure they like "working at big companies" and they're threatening to leave. [*] Your tests take so long that deploying even small fixes is hazardous and slow. Everybody wants to work on something new and fancy but they're stuck debugging "The Rails App."\n\n\n\n\n
Okay, assuming you&#x2019;ve got a Rails app: let&#x2019;s fast forward 3 years from now.\nYou&#x2019;ve done well, you&#x2019;ve got lots of customers and you&#x2019;re up to 20 developers working full-time. Awesome. You&#x2019;re set.\n[*]\nYou now maintain a Rails application running in two datacenters [*] backed by one giant $15K MySQL box in each. [*] It's got 1,000 controllers and models. [*] It's covered by tests but you're not sure how well covered because running RCov would take 7 years. Or SEGFAULT. [*] Your 22 developers complain constantly that it's the worst thing about their job, that they aren't sure they like "working at big companies" and they're threatening to leave. [*] Your tests take so long that deploying even small fixes is hazardous and slow. Everybody wants to work on something new and fancy but they're stuck debugging "The Rails App."\n\n\n\n\n
Okay, assuming you&#x2019;ve got a Rails app: let&#x2019;s fast forward 3 years from now.\nYou&#x2019;ve done well, you&#x2019;ve got lots of customers and you&#x2019;re up to 20 developers working full-time. Awesome. You&#x2019;re set.\n[*]\nYou now maintain a Rails application running in two datacenters [*] backed by one giant $15K MySQL box in each. [*] It's got 1,000 controllers and models. [*] It's covered by tests but you're not sure how well covered because running RCov would take 7 years. Or SEGFAULT. [*] Your 22 developers complain constantly that it's the worst thing about their job, that they aren't sure they like "working at big companies" and they're threatening to leave. [*] Your tests take so long that deploying even small fixes is hazardous and slow. Everybody wants to work on something new and fancy but they're stuck debugging "The Rails App."\n\n\n\n\n
Okay, assuming you&#x2019;ve got a Rails app: let&#x2019;s fast forward 3 years from now.\nYou&#x2019;ve done well, you&#x2019;ve got lots of customers and you&#x2019;re up to 20 developers working full-time. Awesome. You&#x2019;re set.\n[*]\nYou now maintain a Rails application running in two datacenters [*] backed by one giant $15K MySQL box in each. [*] It's got 1,000 controllers and models. [*] It's covered by tests but you're not sure how well covered because running RCov would take 7 years. Or SEGFAULT. [*] Your 22 developers complain constantly that it's the worst thing about their job, that they aren't sure they like "working at big companies" and they're threatening to leave. [*] Your tests take so long that deploying even small fixes is hazardous and slow. Everybody wants to work on something new and fancy but they're stuck debugging "The Rails App."\n\n\n\n\n
{step away. Transition. Pause}\n
{step away. Transition. Pause}\n
If you doubt that this can happen, ask folks at Square. Or Twitter. Or Groupon, LivingSocial, ModCloth, Yellowpages, etc.\nThey've all survived a point at which their Rails app became a terrible burden to their development process. And, from what I&#x2019;ve heard, the problem appeared in roughly the same form each time.\n
If you doubt that this can happen, ask folks at Square. Or Twitter. Or Groupon, LivingSocial, ModCloth, Yellowpages, etc.\nThey've all survived a point at which their Rails app became a terrible burden to their development process. And, from what I&#x2019;ve heard, the problem appeared in roughly the same form each time.\n
If you doubt that this can happen, ask folks at Square. Or Twitter. Or Groupon, LivingSocial, ModCloth, Yellowpages, etc.\nThey've all survived a point at which their Rails app became a terrible burden to their development process. And, from what I&#x2019;ve heard, the problem appeared in roughly the same form each time.\n
If you doubt that this can happen, ask folks at Square. Or Twitter. Or Groupon, LivingSocial, ModCloth, Yellowpages, etc.\nThey've all survived a point at which their Rails app became a terrible burden to their development process. And, from what I&#x2019;ve heard, the problem appeared in roughly the same form each time.\n
If you doubt that this can happen, ask folks at Square. Or Twitter. Or Groupon, LivingSocial, ModCloth, Yellowpages, etc.\nThey've all survived a point at which their Rails app became a terrible burden to their development process. And, from what I&#x2019;ve heard, the problem appeared in roughly the same form each time.\n
If you doubt that this can happen, ask folks at Square. Or Twitter. Or Groupon, LivingSocial, ModCloth, Yellowpages, etc.\nThey've all survived a point at which their Rails app became a terrible burden to their development process. And, from what I&#x2019;ve heard, the problem appeared in roughly the same form each time.\n
Sidebar: this is not a "Rails can't scale" talk. "Can Rails scale" isn't' even a valid question.\n&#x201C;Moar dynos please&#x201D;. Done. You&#x2019;ve just scaled.\n\n\n
Sidebar: this is not a "Rails can't scale" talk. "Can Rails scale" isn't' even a valid question.\n&#x201C;Moar dynos please&#x201D;. Done. You&#x2019;ve just scaled.\n\n\n
There are 4 kinds of scaling that are hard. None of these have anything to do with Rails. Data is a solved problem, it&#x2019;s just hard to implement. Git solves the codebase problem. Scaling your customers is primarily a communications (and logging) problem, and Rails is actually pretty good at helping you scale your features. In fact, that&#x2019;s part of the problem.\n\n
This \n\n
{pause. Transition}\nSo, what IS going wrong with so many of these Rails applications that get giant and are the source of so much pain?\nTo answer that, let's go back to day 1 of the project.\n
In the beginning you had just 1 product and it was implemented by just 1 application. Maybe even just one developer.\n
In the beginning you had just 1 product and it was implemented by just 1 application. Maybe even just one developer.\n
In the beginning you had just 1 product and it was implemented by just 1 application. Maybe even just one developer.\n
You need an admin interface? Awesome. Just build a new '/admin' set of controllers that only certain users can access and query your data in interesting ways.\n[*]\nThe data is always right there, available through ActiveRecord.\n\n
You need an admin interface? Awesome. Just build a new '/admin' set of controllers that only certain users can access and query your data in interesting ways.\n[*]\nThe data is always right there, available through ActiveRecord.\n\n
Need to send another kind of email? You've got an observer on User that triggers an ActionMailer method. Perfect.\n\n
Do users have twitter accounts? Phone numbers? Email addresses? Just add them to the User model.\n\n
All of the common code patterns invented while building the app get extracted into ./lib\n[*]\nand, if you work at a unusual company, the tests are in ./spec/lib.\n
This works GREAT. For about 10,000 users and 10 developers. After that, none of the above is a good idea.\n\n
Rails is tuned to be a rapid prototyping system. \n[*]\nYou can still build something massive out of it, but you have to ignore the defaults because all of Rails' defaults are optimized for the initial application-creation experience.\n
In the beginning you had just 1 product and it was implemented by just 1 application. Maybe even just one developer.\n[*]\nYou need to put new tables into the same database (and database type) because you'll need to do joins across tables, right?\nJoins are great for offline and ad-hoc processing but they're actually pretty bad for many HTTP-backed operations. If you notice how Rails does eager loading it selects rows from one table, saves those ids, and then selects rows from another table.\n[*]\nIf you're already performing these two distinct table queries there's no reason why the second query can't happen in another database or another datastore entirely.There's nothing stopping us from finding a bunch of user ids in our transactional database and then loading their locations, sorted by distance from, say, Riak.\nAs Xavier Shay has been teaching me: Build a software interface to access your data, don&#x2019;t depend on your data living in any specific place.\n\n
In the beginning you had just 1 product and it was implemented by just 1 application. Maybe even just one developer.\n[*]\nYou need to put new tables into the same database (and database type) because you'll need to do joins across tables, right?\nJoins are great for offline and ad-hoc processing but they're actually pretty bad for many HTTP-backed operations. If you notice how Rails does eager loading it selects rows from one table, saves those ids, and then selects rows from another table.\n[*]\nIf you're already performing these two distinct table queries there's no reason why the second query can't happen in another database or another datastore entirely.There's nothing stopping us from finding a bunch of user ids in our transactional database and then loading their locations, sorted by distance from, say, Riak.\nAs Xavier Shay has been teaching me: Build a software interface to access your data, don&#x2019;t depend on your data living in any specific place.\n\n
Don't pick MySQL. There are lots of great features in Postgres but you don't need them, what you DO need is O(1) table alterations for most kinds of 'ALTER TABLE' commands.\nWhat that means is that if you have 10 records, MySQL and Postgres are the same. If you have a billion records, MySQL might take 1 hour and Postgres might take half a second.\nMySQL is officially deprecated at Square.\n{anec: Nolan extending the length of a VARCHAR on a billion rows}\n
Any guesses what you should replace ActionMailer with?\n[*]\nYou just want to send a hash of data to something that can put it into a specific template and deliver it to a given email address. That's no reason in the world that that should happen inside your quickly-bloating Rails app. We should be just POSTing JSON to something that can format, record, and deliver emails.\nThe 400-line redundant singleton mailer class is an eyesore and the views are not web views anyway so they should live in a separate app that caters to that nasty subset of html that can be rendered in an email client.\n
[*]\nThe `users` table should have an id, a bcrypted password or whateveris required for authentication. Nothing that isn't authentication-related should be allowed in this table. It&#x2019;s the easiest - and the worst - place to put new features.\n{anec: string:status -> datetime:updated-at\n\n
Any generic code should be extracted to an internal gem. \n[*]\nIf it doesn't change then it shouldn't be part of your app's code and test suite. This'll speed up your tests, slim down your project size and force you to develop a good interface for this piece of code you've written.\n\n
These are good enough at the start and they're helpful for error messages.\n[*]\nbut you're going to eventually get corrupt data if you don't have db validations. This will make you sad.\n\n
It's not enough. You need to be able to explain to a customer (or at least to yourself) why anything happened in the system.\n[*]\nIf you aren't keeping old versions of your data around then you need to manually add logging around all interesting events.\n{anec: you&#x2019;ve probably got 2-3 half-finished state machines in your app, give them some logging}\n
Rails makes it so easy to add an analytics section to your project but it's such a bad idea. The code is easy to write but your data is in the wrong shape. \n[*]\nSo get a second database and ETL your data over there every day. Once you have more people this analytics component will be a full-time job for somebody and they should have free-reign to write code without cluttering up your primary production app and db.{anecdote about CMD+R}\n\n\n
So the "blog-in-15-minutes" parts of Rails might have lured you into picking a few patterns that start making less sense as you grow.But you&#x2019;re going to have time, eventually to clean things up before they get too bad, right?\nActually, no. The pain will come very suddenly. Sometime between 8 and 30 developers. It'll be the moment you split your one team into multiple teams.\n
An application's structure mimics the organizational structure that produced it; it has to. That's the only way we can make sense of the system we're building.\n{pause}\nQuality work comes from a feeling of ownership. And if your work is fused inappropriately with somebody else's then you can't tell whether you've done a great job. You can't feel the pride of being a professional because somebody else has hacked some marketing feature into your analytics data. Or hacked some analytics system into your marketing code.\n{pause}\n\nSo the moment you switch from a single team to multiple smaller teams the codebase needs to switch, too.\n\n
An application's structure mimics the organizational structure that produced it; it has to. That's the only way we can make sense of the system we're building.\n{pause}\nQuality work comes from a feeling of ownership. And if your work is fused inappropriately with somebody else's then you can't tell whether you've done a great job. You can't feel the pride of being a professional because somebody else has hacked some marketing feature into your analytics data. Or hacked some analytics system into your marketing code.\n{pause}\n\nSo the moment you switch from a single team to multiple smaller teams the codebase needs to switch, too.\n\n
[*]\nIt's something Rails can do (better than most frameworks), but it's not easy.\n
I&#x2019;ve been hearing a term for these monolithic Rails apps. It's "MonoRail".\n\nTo close my talk, and this conference, I'd like to give you the best advice I've gathered from engineers at Square and other companies who've dealt with a MonoRail. My hope is that you, too reach that inflection point where you split into teams and when you hit it you won't miss a beat.\n
The best thing you can do to successfully outgrow a monorail is to interact with different feature sets in your app through some kind of service interface. You pass a little info in, you get a little info back, and you don't touch the code or data on the other side of that wall except through the interface.\nAn example is sending email. Rather than 'ApplicationMailer.deliver_something' you should be writing 'Email.to "some@email.com", "welcome", {:show_greeting => true}'\n\n
Whatever happens on the other side of that, whether it's rendered with ActiveMailer or goes over the wire to some API is no longer your concern.\nOne day, when you split into teams, you can hand everything under that interface to somebody and the lines between "our code" and "their code" is crystal clear.\n
Start with Postgres but get comfortable with other dbs. Add \nRedis, Solr, and some kind of distributed key-value (Riak, etc.) store to your project and start using those systems for appropriate data.\n
Start with Postgres but get comfortable with other dbs. Add \nRedis, Solr, and some kind of distributed key-value (Riak, etc.) store to your project and start using those systems for appropriate data.\n
Start with Postgres but get comfortable with other dbs. Add \nRedis, Solr, and some kind of distributed key-value (Riak, etc.) store to your project and start using those systems for appropriate data.\n
Start with Postgres but get comfortable with other dbs. Add \nRedis, Solr, and some kind of distributed key-value (Riak, etc.) store to your project and start using those systems for appropriate data.\n
Start with Postgres but get comfortable with other dbs. Add \nRedis, Solr, and some kind of distributed key-value (Riak, etc.) store to your project and start using those systems for appropriate data.\n
It becomes hard, after a while, to add new hotness to your product and the thrill is muted. Developers leave big companies because it stops being fun. Or they move to management earlier than they would have.\nAnd this makes perfect sense because you shouldn't spend your finite days here on Earth debugging spaghetti code, explaining your complex system to customers, waiting for a build, or trying in vain to read a hundred thousand lines of a user.rb.\nYou should be spending your days joyfully making new things that delight you and delight the people who pay you. Every day you should walk in to work knowing that you're building the future and not having to maintain the past while your competitors pass you by.\n\n
My name is Jack Danger, I work on the server team at Square, and I thank you for your time.\nAny questions?\n\n\n
