Rails performance at Justin.tv - Guillaume Luccisano

The 7 main actions we took to improvethe Rails stack performance at Justin.tv Guillaume Luccisano - June 2011

Guillaume Luccisano• Work with ruby for about 4 years now• Arrived from France 8 months ago• Ran the migration from Rails 2.1 to 2.3..• ... and to Rails 3 more recently

In the meantine.... I was able to work onimproving performance

I love itAnd I have learn ton of stuff to share with you today

First lessonGuess what?

They lied to usRails 3 is slower than Rails 2

Ok, troll is overSo what’s making our app slow ?

First one is obvious (no) SQL

Less obviousWorker queue wait

External dependencies

Yes, it canMemcached

That one, we love it, so ... Ruby

And its dark sideThe garbage collector

Cool, we have made somegood guesses, now what ?

The Justin.tv Rails stack 12 frontend serversNginx + cache layer with a magic conf from our experts

Talking to 24 app servers

2423: one is currently dead

R.I.P app2Unicorn running with 21 app workers per box 504 workers, yeah!Running Ruby Entreprise Edition with GC tuning

Running at 10% of capacity during normal time We can hit 80K+ Rails requests per minute easily during peak time

One beefy master DB with 7 slaves (not used only by rails)And only 2 memcached boxes for rails

+ some mongo, rabbit, etc... And haproxy everywhereto make us failure resistant...

Cool, so are we ﬁnally going to optimize stuff or not?

Yes, the real ﬁrst step is:Monitoring

You can’t improveperformance in the dark

Newrelic =Awesome

Ganglia =Awesome

Make yourself happy byimproving things and seeing instantly the result

This is the kind of graph we are looking for Still work to do, but it was going down!

Great, we are all setBut eh, what is fast btw?

Fast =Nothing

Less code =Easier to maintain = (often) Faster

I’m curiousWhat is a good average response time? 300ms 200ms 100ms 50ms

we were at 250+msWe are now at 80msAnd there is still ton to do!

How did we do that?

But keep in mind that Every app is unique

1) SQL• Tracked down slow queries, added Indexes• Refactored bad queries • Sometime, 2 queries is faster than a big one• Retrievednetwork and lesscolumns from the db only needed •Less db ruby object creation!

2) C libraries - why?• We added a bunch of C libraries• Bypass ruby memory management • less garbage collection of ruby objects• Raw C speed!• Easy to drop in

2) C libraries - which?• Curb for HTTP (libcurl) (with real timeout) • Support pipelining for mutli requests at the same time• Yajl, the fastest JSON library• Nokogiri, the fastest XML library• Snappy, super fast compression tool by google

Yajl: the facts ~3.5x faster than JSON.generate ~1.9x faster than JSON.parse ~4.5x faster than YAML.load~377.5x faster than YAML.dump ~1.5x faster than Marshal.load ~2x faster than Marshal.dumpAll of this while taking less memory!

Snappy: the factsruby-1.9.2-p180 :061 > f = File.read(Gemfile)ruby-1.9.2-p180 :066 > f.length => 2504GC.start; Benchmark.measure { 1000.times { ActiveSupport::Gzip.compress(f) } }=> 1.840000 0.010000 1.850000 ( 1.842741)GC.start; Benchmark.measure { 1000.times { Snappy.deflate(f) } }=> 0.020000 0.000000 0.020000 ( 0.019659)ruby-1.9.2-p180 :064 > ActiveSupport::Gzip.compress(f).length => 971ruby-1.9.2-p180 :065 > Snappy.deflate(f).length => 1398

3) Memcache• Upgraded our memcached to the last version! • (Not the gem, the real memcached) • We got a x3 improvements!!• Switched everything to the memcached gem • used and made by twitter, use the C libmemcached • 3.5 times faster than Dalli on a simple get (ruby equivalent)

3) Memcache• Used more raw memcache objects • Avoid useless marshal dump • Yajl + Snappy + raw memcache = Win Combo• Removed huge get_multi (100+ items) • It can be slower than the sql query equivalent!• Tuned memcached options

4) Cache expiration• Removed a ton of after_save cache expiration • Using correct expiration time • Or using auto changing cache_key

5) Switched to Unicorn• Like a Boss! Twitter and Github use it.• Fast bootup• Graceful restart• Reduced our queue wait to 0 • Our previous round robin dispatch on our mongrels cluster added up to 40ms delay on average to each request.

6) More GC tuning• Memory vs performance trade off • export RUBY_HEAP_FREE_MIN=100000 • export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 • export RUBY_HEAP_MIN_SLOTS=800000 • export RUBY_HEAP_SLOTS_INCREMENT=200000• We added a GC run after expensive requests: • We divided by 3 our time spent in GC during request

7) Regain memory• Less objects = Faster garbage collection = happiness• Cleaned up our Gemﬁle and removed unused dependencies • aws-s3 gem = 30k ruby objects in your stack • A blank Rails project (2.3 or 3.0) is =~ 100K objects• Cleaned up our codebase! Removing tons of old controllers/views

Regain memory: the facts We refactored our translations system: we saved 50k of useless objects: 10% garbage collection speed upEnough memory saving to add one more unicorn worker

To go further...

• Create or ﬁnd a lighter aws s3 gem! using Curb!• Starting using extra light controller ala Metal for some critical actions• Use snappy to compress fragment caching• Give a try to kiji, ruby fork of REE (from twitter)• Or switch the stack to ruby 1.9 or to jRuby

• Do more memory proﬁling, with tools like memprof• Get a real nonblocking stack to handle several requests per worker • Try Goliath: a non blocking ruby framework• Try the MySQL nosql plugin (if only we were using MySQL!)

Bonus - Extra slidesremoved to save time

Curb: the facts

Memcache - Tune it!• Memcached has a bunch of options: • Auto failover and recovery • Noreply, Noblock • Tcp nodelay, UDP • UDP for set and TCP for get? • Key veriﬁcation • Binary protocol (but slower in ruby, don’t use it :p) • and more.... Play with them!

Clean up your before_ﬁlters We created a speed_up! methodto skip all before_ﬁlters on critical actions speed_up! :only => [‘critical’, ‘action’]

ﬁnd_in_batches

Set.include? instead ofArray.include?

url helpers are slowstore them in a variable when you can to avoid multiple calls

use the bang!like gsub! and avoid new object creation

beware of symbol leakEvery symbol and every string converted to a symbol stay forever in memory => memory leak

The Cloud computing eraCloud is great, but dedicated hardware is still a way faster We monitored a x3 when switching socialcam from h****u to our own cluster.

If you are awesome and want to tackle challenges on awesome products and systems used by millions of users every day: We are currently hiring awesome people http://jobs.justin.tv guillaume@justin.tv Fork me on Github: @kwiRecruiting coordinator at JTV: Brooke (brooke@justin.tv)

Links / References• https://github.com/miyucy/snappy• http://toevolve.org/2011/04/03/http-request-performance.html• https://github.com/brianmario/yajl-ruby• https://github.com/fauna/memcached• http://unicorn.bogomips.org/• http://apidock.com/ruby/Set• http://engineering.twitter.com/2011/05/faster-ruby-kiji-update.html• https://github.com/postrank-labs/goliath• http://blogs.innodb.com/wp/2011/04/nosql-to-innodb-with-memcached/• https://github.com/ice799/memprof

http://www.tricksonrails.com	2814
http://www.ultrasaurus.com	1328
http://feeds.feedburner.com	15
url_unknown	13
http://wix.dapulse.com	11
http://translate.googleusercontent.com	10
http://wix.local.host:3000	6
http://webcache.googleusercontent.com	3
http://paper.li	3
http://www.slideshare.net	2
http://twitter.com	1
http://www.hanrss.com	1
http://www.ethupbolt.appspot.com	1

Rails performance at Justin.tv - Guillaume Luccisano

by Guillaume Luccisano on Jul 01, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

13 Embeds 4,208

Statistics

Rails performance at Justin.tv - Guillaume Luccisano — Presentation Transcript