1) SQL• Tracked down slow queries, added Indexes• Refactored bad queries • Sometime, 2 queries is faster than a big one• Retrievednetwork and lesscolumns from the db only needed •Less db ruby object creation!
2) C libraries - why?• We added a bunch of C libraries• Bypass ruby memory management • less garbage collection of ruby objects• Raw C speed!• Easy to drop in
2) C libraries - which?• Curb for HTTP (libcurl) (with real timeout) • Support pipelining for mutli requests at the same time• Yajl, the fastest JSON library• Nokogiri, the fastest XML library• Snappy, super fast compression tool by google
Yajl: the facts ~3.5x faster than JSON.generate ~1.9x faster than JSON.parse ~4.5x faster than YAML.load~377.5x faster than YAML.dump ~1.5x faster than Marshal.load ~2x faster than Marshal.dumpAll of this while taking less memory!
3) Memcache• Upgraded our memcached to the last version! • (Not the gem, the real memcached) • We got a x3 improvements!!• Switched everything to the memcached gem • used and made by twitter, use the C libmemcached • 3.5 times faster than Dalli on a simple get (ruby equivalent)
3) Memcache• Used more raw memcache objects • Avoid useless marshal dump • Yajl + Snappy + raw memcache = Win Combo• Removed huge get_multi (100+ items) • It can be slower than the sql query equivalent!• Tuned memcached options
4) Cache expiration• Removed a ton of after_save cache expiration • Using correct expiration time • Or using auto changing cache_key
5) Switched to Unicorn• Like a Boss! Twitter and Github use it.• Fast bootup• Graceful restart• Reduced our queue wait to 0 • Our previous round robin dispatch on our mongrels cluster added up to 40ms delay on average to each request.
6) More GC tuning• Memory vs performance trade off • export RUBY_HEAP_FREE_MIN=100000 • export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 • export RUBY_HEAP_MIN_SLOTS=800000 • export RUBY_HEAP_SLOTS_INCREMENT=200000• We added a GC run after expensive requests: • We divided by 3 our time spent in GC during request
7) Regain memory• Less objects = Faster garbage collection = happiness• Cleaned up our Gemﬁle and removed unused dependencies • aws-s3 gem = 30k ruby objects in your stack • A blank Rails project (2.3 or 3.0) is =~ 100K objects• Cleaned up our codebase! Removing tons of old controllers/views
Regain memory: the facts We refactored our translations system: we saved 50k of useless objects: 10% garbage collection speed upEnough memory saving to add one more unicorn worker
• Create or ﬁnd a lighter aws s3 gem! using Curb!• Starting using extra light controller ala Metal for some critical actions• Use snappy to compress fragment caching• Give a try to kiji, ruby fork of REE (from twitter)• Or switch the stack to ruby 1.9 or to jRuby
• Do more memory proﬁling, with tools like memprof• Get a real nonblocking stack to handle several requests per worker • Try Goliath: a non blocking ruby framework• Try the MySQL nosql plugin (if only we were using MySQL!)
Memcache - Tune it!• Memcached has a bunch of options: • Auto failover and recovery • Noreply, Noblock • Tcp nodelay, UDP • UDP for set and TCP for get? • Key veriﬁcation • Binary protocol (but slower in ruby, don’t use it :p) • and more.... Play with them!
Clean up your before_ﬁlters We created a speed_up! methodto skip all before_ﬁlters on critical actions speed_up! :only => [‘critical’, ‘action’]
If you are awesome and want to tackle challenges on awesome products and systems used by millions of users every day: We are currently hiring awesome people http://jobs.justin.tv firstname.lastname@example.org Fork me on Github: @kwiRecruiting coordinator at JTV: Brooke (email@example.com)