Rails performance: Ruby GC tweaking

Rails performance: Ruby GC tweaking Vlad ZLOTEANU @vladzloteanu Software engineer @ Dimelo

What we develop
Frontend products
Collaborative platforms, ‘forum-like’, white-labeled, for big accounts (a kind of GetSatisfaction / UserVoice, but.. Better ;) )
Backend product
SocialAPIs client
(a kind of TweetDeck, but for multiple channels, and designed for teams/multiple users per account)

Stats on Dynamic requests 20 req/s 10 req/s Web Peaks of 40 req/s Peaks of 25 req/s APIs
on HTML/XHTML dynamic requests (we don’ t count CSS, JS, images, etc)

Problem Medium response time ~ 400 ms WTF ??

Ruby perf toolbox Profile Benchmark Server logs Apache Benchmark (ab, ab2) Rack::Bug RailsBench RubyProf RBTrace Memprof Scrap BleakHouse .. Valgrind ltrace Std surveillance Cacti Nagios Surveilance & profiling NewRelic RPM UnionStation

Finding the problem(s)
DB queries (unoptimized queries, n+1)
External services (EG external RSS feeds)
Lots of partials, low or inexistent cache
Slow session storage
Memory issues
Unoptimized GC
Memory leaks

Memory / GC issues
How do I know it’s a memory/GC issue?
Server logs
Profilers (memprof)
Monitorization services (newrelic)

Ruby Garbage Collector
Conservative
Stop-the-world
Mark-and-sweep process

Ruby default GC settings
RUBY_HEAP_MIN_SLOTS = 10000
Initial number of heap slots. It also represents the minimum number of slots, at all times
a new rails app boots up with almost 500k objects on the heap(mostly nodes)
RUBY_HEAP_SLOTS_INCREMENT = 10000
The number of new slots to allocate when all initial slots are used
RUBY_HEAP_SLOTS_GROWTH_FACTOR = 1.8
Next time Ruby needs new heap slots it will use a multiplicator
New stab is almost double sized

Ruby default GC settings (2)
RUBY_GC_MALLOC_LIMIT =8000000
force GC after 8 MB malloc/realloc
High traffic Rails servers can easily allocate more than 8 MB per request
RUBY_HEAP_FREE_MIN =4096
The number of free slots that should be present after GC finishes running. If there are fewer slots than those defined it will allocate new ones

Find your numbers
Start with enough memory to hold Rails (Ruby’s default is practically nothing)
Increase it linearly if you need more (Ruby’s default is exponential increase)
Only garbage-collect every XX million malloc calls (Ruby’s default is way too small)
Benchmark, benchmark, benchmark..

GitHub numbers
RUBY_HEAP_MIN_SLOTS=800000 (10000)
RUBY_HEAP_FREE_MIN=100000 (4096)
RUBY_HEAP_SLOTS_INCREMENT=300000(10000)
RUBY_HEAP_SLOTS_GROWTH_FACTOR=1(1.8)
RUBY_GC_MALLOC_LIMIT=79000000(8000000)

Demo (using ApacheBench) Before optimisations: After optimisations: Execution time(ms): 228 344 322.0 316 4797 Memory usage: 130 MB Execution time(ms): 120 181 190.0 150 2256 Memory usage: 160 MB
Frontend app, 15 requests, concurrency 1

Conclusions
use REE on prod (or Ruby 1.9)
measure, measure, measure (don’ t forget to GC.enable_stats )
optimise what matters
And yes, 5 lines of code can make a difference ;)

Thank you!
Questions?

Rails performance: Ruby GC tweaking

by Dimelo R&D Team on Jan 28, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Rails performance: Ruby GC tweaking — Presentation Transcript