Scaling out Rails with MySQL

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

more share options
Embed For WordPress.com

Without related presentations

Hi, I’m Matt Freels, first, a little bit about me:

- I work at Serious Business over in San Francisco. We make awesome social games on Facebook.
- Notably Friends for Sale, Rock Legends, and Happy Hour.
- If you have been bought or sold by somebody on Facebook, it was probably our fault.
- We are Rails shop (or I probably wouldn’t be here!)
- Our apps collectively handle about 600k daus, millions of pageviews/day

- We need Ruby devs!
- Ping me later

- My presentation has two parts
- I’m going to talk a bit about how we have handled scaling our apps with MySQL...

- And then dig a bit deeper into db migrations and Table Migrator,
- a library i wrote to do ALTER TABLEs on a production app with as little downtime as possible.

- So, part 1
- I figured we could talk take a little time to define the problem itself.

- If a system can scale, it means that its behavior remains constant regardless of its size.
- the more resources you throw at it, proportionally, the more load it can handle.
- If i double the number of servers, my scalable system should be able handle twice the number of users.

- So, usually we start with a small system and then we try to make it bigger
- Which leads to the problem...

- Will it scale?

- The answer lies in concurrency.
- The more segments of your app that don’t care about each other the better.

- As app developers, I think we already do this to an extent.
- Logical segments of our apps are encapsulated.
- Bring this out to the app environment, too.
- Use separate databases for different kinds of data.
- Use separate services for different purposes (like sphinx for search).
- This gets you very far. You might be able to stop here.

- Sharding!
- It can be hard or easy depending on the nature of your data.
- Is your app like a bunch of little villages, each one fairly independent? (sharding is easy)
- Is your app like one big city (here be dragons)

- Make your app fast and make your life easier. If your app is just simply fast, that means you can:
- Scale more easily (fewer moving parts, not to mention fewer expensive servers...)
- Not need to be scalable at all. (The best problem is no problem!)

- So, number one takeaway...

- Use your db wisely.
- Spend some time optimizing how you use your db.
- Optimize indexes, queries.
- Understanding the performance behavior and quirks of MySQL is extremely useful!
- We’ve gotten more bang for the buck doing this than anything else, except for caching.

- The best book I’ve ever read (hyperbole :P ) was High Performance MySQL.

- Like I mentioned before, you want to find ways to divide up your app by function.
- Rather than letting MySQL be your catchall persistent store
- split your monolithic database into multiple ones by purpose
- for us: app content, user-generated content, transient data
- if you don’t need transactions between two models, they can live in separate dbs.
- delegate functionality to more specialized components.

- Rails is your glue code, like your VC in MVC.

- Functional Partitioning is easy. It’s actually a one-liner

- MySQL has a query cache, but it’s inflexible.
- You can use memcache, or something else, as long as it’s quick.

- Cache your models!
- cache_fu and cache_money are libs i know of.
- Model caching is better than fragment or view caching.
- The model is in control
- Simple controller/view logic.
- Same cached data can be used in more places.
- Your cache can provide a slightly different view
- Cache a comments inline on an article for instance.

- Obviously good for full-text search.
- Also good for pivoting through datasets.
- More flexible and more powerful than MySQL’s built-in indexes
- Downside is that up-to-date indexes are hard
- You don’t add to a sphinx index, you have to rebuild it.
- There are workarounds, though.

- Before we were able to do this, we went through a lot of trouble guarding ourselves against schema change in the future and working around mistakes in the past.
- Now, we can just fix the problem.

- What’s the point?

- So, what MySQL gives you when you want to modify your table schema is ALTER TABLE.

- It is synchronous. You can read data, but is writes are stalled.
- It is slow. For everything but a table rename, alter table creates a new table with the new properties, and copies -- line by line -- every row from the old table into the new table.
- That's it! No Magic!
- Obviously, these two facts taken together with large tables...

is awesome.
- (Hi, welcome to our website! We'll be back in 10 hours!)

- We can do better
- It would be nice it was faster overall, but the big problem is the time spent in that table lock.
- MySQL takes the easy way out to keep the old table and new table in sync. It just doesn’t let you touch them.
- However, if we keep track of changes, we can allow access to the old table while copying its data to the new one, and then in a table lock, all we have to do is copy over what has changed, and swap in the new table.
-

- The bottleneck at this point is how long it takes to find all of the changed rows and copy them over.
- We can improve on this, though, if we make multiple asynchronous sweeps to pull over changes, the result of each being that the delta between the two tables is smaller each time.

- With this, the bottleneck is finding those modified rows. If you have an index on the column you are using to do this...

- you can continue to sweep in changes until only a relatively few rows are left that need to be copied over during a final synchronous copy-swap phase.
- Caveat: There is a hole where the table doesn't exist for a moment. Ideas?

Favorites, Groups & Events

Scaling out Rails with MySQL - Presentation Transcript

Hi
We’re hiring!
Part I: Scaling with MySQL
Part II: Table Migrator
Scale What is it?
More is more 3x resources = 3x capacity
Will it scale?
Concurrency is key
Functional Partitioning
Horizontal Partitioning The dirty word: Sharding
Back to MySQL Protips :P
Scaling vs. Performance
Optimize!
Embrace and Extend
Partitioning
Partitioning class NewsStory < ActiveRecord::Base establish_connection 'development_news_stories' ... end
Caching
Model Caching
Model Caching cache_fu cache_money
Sphinx
Part II: Table Migrator
How it works:
ALTER TABLE
ALTER TABLE Full of Win
ALTER TABLE Time Taken Locked (a long freakin’ time)
Doing it Better
ALTER TABLE Optimized Unlocked Locked (a shorter time)
ALTER TABLE Optimized Unlocked Locked (a shorter time)
ALTER TABLE Optimized Unlocked Locked (milliseconds)
Demo
That’s it! Thanks! http://matt.freels.name http://github.com/freels/table_migrator