Scaling Twitter

Scaling Twitter - Presentation Transcript

Big Bird. (scaling twitter)

Rails Scales. (but not out of the box)

First, Some Facts • 600 requests per second. Growing fast. • 180 Rails Instances (Mongrel). Growing fast. • 1 Database Server (MySQL) + 1 Slave. • 30-odd Processes for Misc. Jobs • 8 Sun X4100s • Many users, many updates.

Joy Pain Oct Nov Dec Jan Feb March Apr

IM IN UR RAILZ MAKIN EM GO FAST

It’s Easy, Really. 1. Realize Your Site is Slow 2. Optimize the Database 3. Cache the Hell out of Everything 4. Scale Messaging 5. Deal With Abuse

It’s Easy, Really. 1. Realize Your Site is Slow 2. Optimize the Database 3. Cache the Hell out of Everything 4. Scale Messaging 5. Deal With Abuse 6. Proﬁt

the more you know { Part the First }

We Failed at This.

Don’t Be Like Us • Munin • Nagios • AWStats & Google Analytics • Exception Notiﬁer / Exception Logger • Immediately add reporting to track problems.

Test Everything • Start Before You Start • No Need To Be Fancy • Tests Will Save Your Life • Agile Becomes Important When Your Site Is Down

Benchmarks? let your users do it.

The Database { Part the Second }

“The Next Application I Build is Going to Be Easily Partitionable” - S. Butterﬁeld

Too Late.

Index Everything

class AddIndex < ActiveRecord::Migration def self.up add_index :users, :email end def self.down remove_index :users, :email end end Repeat for any column that appears in a WHERE clause Rails won’t do this for you.

Denormalize A Lot

class DenormalizeFriendsIds < ActiveRecord::Migration def self.up add_column "users", "friends_ids", :text end def self.down remove_column "users", "friends_ids" end end

class Friendship < ActiveRecord::Base belongs_to :user belongs_to :friend after_create :add_to_denormalized_friends after_destroy :remove_from_denormalized_friends def add_to_denormalized_friends user.friends_ids << friend.id user.friends_ids.uniq! user.save_without_validation end def remove_from_denormalized_friends user.friends_ids.delete(friend.id) user.save_without_validation end end

Don’t be Stupid

bob.friends.map(&:email) Status.count() “email like ‘%#{search}%’”

That’s where we are. Seriously. If your Rails application is doing anything more complex than that, you’re doing something wrong*. * or you observed the First Rule of Butterﬁeld.

Partitioning Comes Later. (we’ll let you know how it goes)

The Cache { Part the Third }

MemCache

!

class Status < ActiveRecord::Base class << self def count_with_memcache(*args) return count_without_memcache unless args.empty? count = CACHE.get(“status_count”) if count.nil? count = count_without_memcache CACHE.set(“status_count”, count) end count end alias_method_chain :count, :memcache end after_create :increment_memcache_count after_destroy :decrement_memcache_count ... end

class User < ActiveRecord::Base def friends_statuses ids = CACHE.get(“friends_statuses:#{id}”) Status.find(:all, :conditions => [“id IN (?)”, ids]) end end class Status < ActiveRecord::Base after_create :update_caches def update_caches user.friends_ids.each do |friend_id| ids = CACHE.get(“friends_statuses:#{friend_id}”) ids.pop ids.unshift(id) CACHE.set(“friends_statuses:#{friend_id}”, ids) end end end

The Future ve d ti r co Ac e R

90% API Requests Cache Them!

“There are only two hard things in CS: cache invalidation and naming things.” – Phil Karlton, via Tim Bray

Messaging { Part the Fourth }

You Already Knew All That Other Stuff, Right?

Producer Consumer Message Producer Consumer Queue Producer Consumer

DRb • The Good: • Stupid Easy • Reasonably Fast • The Bad: • Kinda Flaky • Zero Redundancy • Tightly Coupled

ejabberd Jabber Client (drb) Incoming Outgoing Presence Messages Messages MySQL

Server DRb.start_service ‘druby://localhost:10000’, myobject Client myobject = DRbObject.new_with_uri(‘druby://localhost:10000’)

Rinda • Shared Queue (TupleSpace) • Built with DRb • RingyDingy makes it stupid easy • See Eric Hodel’s documentation • O(N) for take(). Sigh.

Timestamp: 12/22/06 01:53:14 (4 months ago) Author: lattice Message: Fugly. Seriously. Fugly. SELECT * FROM messages WHERE substring(truncate(id,0),-2,1) = #{@fugly_dist_idx}

It Scales. (except it stopped on Tuesday)

Options • ActiveMQ (Java) • RabbitMQ (erlang) • MySQL + Lightweight Locking • Something Else?

erlang? What are you doing? Stabbing my eyes out with a fork.

Starling • Ruby, will be ported to something faster • 4000 transactional msgs/s • First pass written in 4 hours • Speaks MemCache (set, get)

Use Messages to Invalidate Cache (it’s really not that hard)

Abuse { Part the Fifth }

The Italians

9000 friends in 24 hours (doesn’t scale)

http://flickr.com/photos/heather/464504545/ http://flickr.com/photos/curiouskiwi/165229284/ http://flickr.com/photo_zoom.gne?id=42914103&size=l http://flickr.com/photos/madstillz/354596905/ http://flickr.com/photos/laughingsquid/382242677/ http://flickr.com/photos/bng/46678227/