Scaling Twitter

Slide 1: Big Bird. (scaling twitter)

Slide 2: Rails Scales. (but not out of the box)

Slide 3: First, Some Facts • 600 requests per second. Growing fast. • 180 Rails Instances (Mongrel). Growing fast. • 1 Database Server (MySQL) + 1 Slave. • 30-odd Processes for Misc. Jobs • 8 Sun X4100s • Many users, many updates.

Slide 7: Joy Pain Oct Nov Dec Jan Feb March Apr

Slide 8: IM IN UR RAILZ MAKIN EM GO FAST

Slide 9: It’s Easy, Really. 1. Realize Your Site is Slow 2. Optimize the Database 3. Cache the Hell out of Everything 4. Scale Messaging 5. Deal With Abuse

Slide 10: It’s Easy, Really. 1. Realize Your Site is Slow 2. Optimize the Database 3. Cache the Hell out of Everything 4. Scale Messaging 5. Deal With Abuse 6. Profit

Slide 11: the more you know { Part the First }

Slide 12: We Failed at This.

Slide 13: Don’t Be Like Us • Munin • Nagios • AWStats & Google Analytics • Exception Notifier / Exception Logger • Immediately add reporting to track problems.

Slide 14: Test Everything • Start Before You Start • No Need To Be Fancy • Tests Will Save Your Life • Agile Becomes Important When Your Site Is Down

Slide 15: <!-- served to you through a copper wire by sampaati at 22 Apr 15:02 in 343 ms (d 102 / r 217). thank you, come again. --> <!-- served to you through a copper wire by kolea.twitter.com at 22 Apr 15:02 in 235 ms (d 87 / r 130). thank you, come again. --> <!-- served to you through a copper wire by raven.twitter.com at 22 Apr 15:01 in 450 ms (d 96 / r 337). thank you, come again. --> Benchmarks? let your users do it. <!-- served to you through a copper wire by kolea.twitter.com at 22 Apr 15:00 in 409 ms (d 88 / r 307). thank you, come again. --> <!-- served to you through a copper wire by firebird at 22 Apr 15:03 in 2094 ms (d 643 / r 1445). thank you, come again. --> <!-- served to you through a copper wire by quetzal at 22 Apr 15:01 in 384 ms (d 70 / r 297). thank you, come again. -->

Slide 16: The Database { Part the Second }

Slide 17: “The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield

Slide 18: “The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield

Slide 19: “The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield

Slide 20: Too Late.

Slide 21: Index Everything

Slide 22: class AddIndex < ActiveRecord::Migration def self.up add_index :users, :email end def self.down remove_index :users, :email end end Repeat for any column that appears in a WHERE clause Rails won’t do this for you.

Slide 23: Denormalize A Lot

Slide 24: class DenormalizeFriendsIds < ActiveRecord::Migration def self.up add_column "users", "friends_ids", :text end def self.down remove_column "users", "friends_ids" end end

Slide 25: class Friendship < ActiveRecord::Base belongs_to :user belongs_to :friend after_create :add_to_denormalized_friends after_destroy :remove_from_denormalized_friends def add_to_denormalized_friends user.friends_ids << friend.id user.friends_ids.uniq! user.save_without_validation end def remove_from_denormalized_friends user.friends_ids.delete(friend.id) user.save_without_validation end end

Slide 26: Don’t be Stupid

Slide 27: bob.friends.map(&:email) Status.count() “email like ‘%#{search}%’”

Slide 28: That’s where we are. Seriously. If your Rails application is doing anything more complex than that, you’re doing something wrong*. * or you observed the First Rule of Butterfield.

Slide 29: Partitioning Comes Later. (we’ll let you know how it goes)

Slide 30: The Cache { Part the Third }

Slide 31: MemCache

Slide 32: MemCache

Slide 33: MemCache

Slide 34: !

Slide 35: class Status < ActiveRecord::Base class << self def count_with_memcache(*args) return count_without_memcache unless args.empty? count = CACHE.get(“status_count”) if count.nil? count = count_without_memcache CACHE.set(“status_count”, count) end count end alias_method_chain :count, :memcache end after_create :increment_memcache_count after_destroy :decrement_memcache_count ... end

Slide 36: class User < ActiveRecord::Base def friends_statuses ids = CACHE.get(“friends_statuses:#{id}”) Status.find(:all, :conditions => [“id IN (?)”, ids]) end end class Status < ActiveRecord::Base after_create :update_caches def update_caches user.friends_ids.each do |friend_id| ids = CACHE.get(“friends_statuses:#{friend_id}”) ids.pop ids.unshift(id) CACHE.set(“friends_statuses:#{friend_id}”, ids) end end end

Slide 37: The Future ve d ti r co Ac e R

Slide 38: 90% API Requests Cache Them!

Slide 39: “There are only two hard things in CS: cache invalidation and naming things.” – Phil Karlton, via Tim Bray

Slide 40: Messaging { Part the Fourth }

Slide 41: You Already Knew All That Other Stuff, Right?

Slide 42: Producer Consumer Message Producer Consumer Queue Producer Consumer

Slide 43: DRb • The Good: • Stupid Easy • Reasonably Fast • The Bad: • Kinda Flaky • Zero Redundancy • Tightly Coupled

Slide 44: ejabberd Jabber Client (drb) Incoming Outgoing Presence Messages Messages MySQL

Slide 45: Server DRb.start_service ‘druby://localhost:10000’, myobject Client myobject = DRbObject.new_with_uri(‘druby://localhost:10000’)

Slide 46: Rinda • Shared Queue (TupleSpace) • Built with DRb • RingyDingy makes it stupid easy • See Eric Hodel’s documentation • O(N) for take(). Sigh.

Slide 47: Timestamp: 12/22/06 01:53:14 (4 months ago) Author: lattice Message: Fugly. Seriously. Fugly. SELECT * FROM messages WHERE substring(truncate(id,0),-2,1) = #{@fugly_dist_idx}

Slide 48: It Scales. (except it stopped on Tuesday)

Slide 49: Options • ActiveMQ (Java) • RabbitMQ (erlang) • MySQL + Lightweight Locking • Something Else?

Slide 50: erlang? What are you doing? Stabbing my eyes out with a fork.

Slide 51: Starling • Ruby, will be ported to something faster • 4000 transactional msgs/s • First pass written in 4 hours • Speaks MemCache (set, get)

Slide 52: Use Messages to Invalidate Cache (it’s really not that hard)

Slide 53: Abuse { Part the Fifth }

Slide 54: The Italians

Slide 55: 9000 friends in 24 hours (doesn’t scale)

Slide 56: http://flickr.com/photos/heather/464504545/ http://flickr.com/photos/curiouskiwi/165229284/ http://flickr.com/photo_zoom.gne?id=42914103&size=l http://flickr.com/photos/madstillz/354596905/ http://flickr.com/photos/laughingsquid/382242677/ http://flickr.com/photos/bng/46678227/