Debugging Ruby
troubleshooting the VM and your app
Aman Gupta

About Aman Gupta
San Francisco, CA
Ruby Performance Consulting
(Twitter, Github, Heroku, ZenDesk)
Ruby Hero 2009
EventMachine, amqp, REE, sinbook,
perftools.rb, gdb.rb
github.com/tmm1
@tmm1

The Ruby VM
MRI- Ruby 1.8.{6,7}
REE- Ruby 1.8.7 + patches
Tools for C code
Tested primarily on:
Linux operating system
Intel CPUs (i686 and x86_64)

lsof
list open files
lsof -nPp <pid>

lsof -nPp <pid>
-n
Inhibits the conversion of network numbers to host names for
network files.
-P
Inhibits the conversion of port numbers to port names for network
files
FD TYPE SIZE NODE NAME
json
cwd DIR 4096 10437557 /var/www/myapp memcached
rtd DIR 4096 2 /
mysql
txt REG 1925210 1089684 /usr/bin/ruby
mem REG 48188 7061523 /json-1.1.9/ext/json/ext/generator.so http
mem REG 46970 7061524 /json-1.1.9/ext/json/ext/parser.so
mem REG 3428025 1131339 /memcached-0.17.4/lib/rlibmemcached.so
mem REG 152972 5303443 /mysql-2.8.1/lib/mysql_api.so
mem REG 154013 1089708 /usr/lib/libtcmalloc_minimal.so.0.0.0
mem REG 119288 11616294 /lib/ld-2.7.so
0u CHR 303 /dev/null
1w REG 4746405529 1106245 /usr/local/nginx/logs/error.log
2w REG 4746405529 1106245 /usr/local/nginx/logs/error.log
3u IPv4 TCP 10.8.85.66:33326->10.8.85.68:3306 (ESTABLISHED)
10u IPv4 TCP 10.8.85.66:33327->10.8.85.68:3306 (ESTABLISHED)
11u IPv4 TCP 127.0.0.1:58273->127.0.0.1:11211 (ESTABLISHED)
12u REG 20182 12463107 /tmp/RackMultipart.28957.0
33u IPv4 TCP 174.36.83.42:37466->69.63.180.21:80 (ESTABLISHED)

strace
trace system calls and signals
strace -cp <pid>
strace -ttTp <pid> -o <file>

strace -cp <pid>
-c
Count time, calls, and errors for each system call and report a
summary on program exit.
-p pid
Attach to the process with the process ID pid and begin tracing.
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
50.39 0.000064 0 1197 592 read
34.65 0.000044 0 609 writev
14.96 0.000019 0 1226 epoll_ctl
0.00 0.000000 0 4 close
0.00 0.000000 0 1 select
0.00 0.000000 0 4 socket
0.00 0.000000 0 4 4 connect
0.00 0.000000 0 1057 epoll_wait
------ ----------- ----------- --------- --------- ----------------
100.00 0.000127 4134 596 total

strace -ttTp <pid> -o <file>
-t
Prefix each line of the trace with the time of day.
-tt
If given twice, the time printed will include the microseconds.
-T
Show the time spent in system calls. This records the time
difference between the beginning and the end of each system call.
-o filename
Write the trace output to the file filename rather than to stderr.
01:09:11.266949 epoll_wait(9, {{EPOLLIN, {u32=68841296, u64=68841296}}}, 4096, 50) = 1 <0.033109>
01:09:11.300102 accept(10, {sa_family=AF_INET, sin_port=38313, sin_addr="127.0.0.1"}, [1226]) = 22 <0.000014>
01:09:11.300190 fcntl(22, F_GETFL) = 0x2 (flags O_RDWR) <0.000007>
01:09:11.300237 fcntl(22, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000008>
01:09:11.300277 setsockopt(22, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>
01:09:11.300489 accept(10, 0x7fff5d9c07d0, [1226]) = -1 EAGAIN <0.000014>
01:09:11.300547 epoll_ctl(9, EPOLL_CTL_ADD, 22, {EPOLLIN, {u32=108750368, u64=108750368}}) = 0 <0.000009>
01:09:11.300593 epoll_wait(9, {{EPOLLIN, {u32=108750368, u64=108750368}}}, 4096, 50) = 1 <0.000007>
01:09:11.300633 read(22, "GET / HTTP/1.1r"..., 16384) = 772 <0.000012>
01:09:11.301727 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000007>
01:09:11.302095 poll([{fd=5, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout) <0.000008>
01:09:11.302144 write(5, "1000000-0003SELECT * FROM `table`"..., 56) = 56 <0.000023>
01:09:11.302221 read(5, "25101,20x234m"..., 16384) = 284 <1.300897>

stracing ruby: SIGVTALRM
15:45:51.658164 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
15:45:51.658244 rt_sigreturn(0x1a) = 2207807 <0.000009>
15:45:51.678208 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
15:45:51.678271 rt_sigreturn(0x1a) = 0 <0.000009>
15:45:51.698161 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
15:45:51.698216 rt_sigreturn(0x1a) = 140734552062624 <0.000009>
15:45:51.718154 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
15:45:51.718192 rt_sigreturn(0x1a) = 140734552066688 <0.000009>
15:45:51.738185 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
15:45:51.738221 rt_sigreturn(0x1a) = 11333952 <0.000008>
15:45:51.758162 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
15:45:51.758216 rt_sigreturn(0x1a) = 0 <0.000009>
15:45:51.818168 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
15:45:51.819817 rt_sigreturn(0x1a) = 1 <0.000010>
15:45:51.838196 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
ruby 1.8 uses signals to schedule its green threads
process receives a SIGVTALRM signal every 10ms

stracing ruby: sigprocmask
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.326334 0 3568567 rt_sigprocmask
0.00 0.000000 0 9 read
0.00 0.000000 0 10 open
0.00 0.000000 0 10 close
0.00 0.000000 0 9 fstat
0.00 0.000000 0 25 mmap
------ ----------- ----------- --------- --------- ----------------
100.00 0.326334 3568685 0 total
debian/redhat compile ruby with --enable-pthread
uses a native thread timer for SIGVTALRM
causes excessive calls to sigprocmask: 30% slowdown!

tcpdump
dump traffic on a network
tcpdump -i eth1 -s 1500 -nqA
tcp dst port 80
tcpdump -i eth0 -s 1500 -nqA
tcp dst port 3306
tcpdump -i eth1 -s 1500 -w <file>
tcp dst port 80

tcpdump -i <eth> -s <len> -nqA <expr>
tcpdump -i <eth> -w <file> <expr>
-i <eth>
Network interface.
-s <len>
Snarf len bytes of data from each packet.
-n
Don't convert addresses (host addresses, port numbers) to names.
-q
Quiet output. Print less protocol information.
-A
Print each packet (minus its link level header) in ASCII.
-w <file>
Write the raw packets to file rather than printing them out.
<expr>
libpcap expression, for example:
tcp src port 80
tcp dst port 3306

tcp dst port 80
19:52:20.216294 IP 24.203.197.27.40105 > 174.37.48.236.80: tcp 438
E...*.@.l.%&.....%0....POx..%s.oP.......GET /poll_images/cld99erh0/logo.png HTTP/1.1
Accept: */*
Referer: http://apps.facebook.com/realpolls/?_fb_q=1
tcp dst port 3306
19:51:06.501632 IP 10.8.85.66.50443 > 10.8.85.68.3306: tcp 98
E..."K@.@.Yy
.UB
.UD.....z....L............
GZ.y3b..[......W....SELECT * FROM `votes` WHERE (`poll_id` = 72621) LIMIT 1
tcpdump -w <file>

google-perftools
Google’s CPU profiler
export LD_PRELOAD=libprofiler.so
export DYLD_INSERT_LIBRARIES=libprofiler.dylib
CPUPROFILE=/tmp/myprof ./myapp
pprof ./myapp /tmp/myprof

wget http://google-perftools.googlecode.com/files/google-
perftools-1.4.tar.gz download
tar zxvf google-perftools-1.4.tar.gz
cd google-perftools-1.4
./configure --prefix=/opt
make compile
sudo make install
# for linux
export LD_PRELOAD=/opt/lib/libprofiler.so setup
# for osx
export DYLD_INSERT_LIBRARIES=/opt/lib/libprofiler.dylib
CPUPROFILE=/tmp/ruby.prof ruby -e' profile
5_000_000.times{ "hello world" }
'
pprof `which ruby` --text /tmp/ruby.prof report

pprof ruby pprof ruby
ruby.prof --text ruby.prof --gif
Total: 103 samples
20 19.4% 19.4% 95 92.2% rb_yield_0
11 10.7% 30.1% 103 100.0% rb_eval
8 7.8% 37.9% 12 11.7% gc_sweep
3 2.9% 68.9% 52 50.5% rb_str_new3
3 2.9% 74.8% 3 2.9% obj_free
3 2.9% 77.7% 103 100.0% int_dotimes
3 2.9% 80.6% 12 11.7% gc_mark

Profiling MRI
10% of production VM
time spent in
rb_str_sub_bang
String#sub!
called from Time.parse
return unless str.sub!(/A(d{1,2})/, '')
return unless str.sub!(/A( d|d{1,2})/, '')
return unless str.sub!(/A( d|d{1,2})/, '')
return unless str.sub!(/A(d{1,3})/, '')
return unless str.sub!(/A(d{1,2})/, '')
return unless str.sub!(/A(d{1,2})/, '')
switch to third_base gem
ThirdBase: Fast and Easy
Date/DateTime class for
Ruby

Profiling EM + threads
Total: 3763 samples
2764 73.5% catch_timer
989 26.3% memcpy
3 0.1% st_lookup
2 0.1% rb_thread_schedule
1 0.0% rb_eval
1 0.0% rb_newobj
1 0.0% rb_gc_force_recycle
known issue: EM+threads =
slow
memcpy??
thread context switches
copy the stack w/ memcpy
EM allocates huge buffer
on the stack
solution: move buffer to
the heap

ltrace
trace library calls
ltrace -cp <pid>
ltrace -ttTp <pid> -o <file>

ltrace -c ruby threaded_em.rb
% time seconds usecs/call calls function
------ ----------- ----------- --------- --------------------
48.65 11.741295 617 19009 memcpy
30.16 7.279634 831 8751 longjmp
9.78 2.359889 135 17357 _setjmp
8.91 2.150565 285 7540 malloc
1.10 0.265946 20 13021 memset
0.81 0.195272 19 10105 __ctype_b_loc
0.35 0.084575 19 4361 strcmp
0.19 0.046163 19 2377 strlen
0.03 0.006272 23 265 realloc
------ ----------- ----------- --------- --------------------
100.00 24.134999 82999 total
ltrace -ttT -e memcpy ruby threaded_em.rb
01:24:48.769408 --- SIGVTALRM (Virtual timer expired) ---
01:24:48.769616 memcpy(0x1216000, "", 1086328) = 0x1216000 <0.000578>
01:24:48.770555 memcpy(0x6e32670, "240&343v", 1086328) = 0x6e32670 <0.000418>
01:24:49.899414 --- SIGVTALRM (Virtual timer expired) ---
01:24:49.899490 memcpy(0x1320000, "", 1082584) = 0x1320000 <0.000628>
01:24:49.900474 memcpy(0x6e32670, "", 1086328) = 0x6e32670 <0.000479>

ltrace/libdl
trace dlopen’d library calls
ltrace -F <conf> -bg -x <symbol>
-p <pid>
http://github.com/ice799/ltrace/tree/libdl

ltrace -F <conf> -b -g -x <sym>
-b
Ignore signals.
-g
Ignore libraries linked at compile time.
-F <conf>
Read prototypes from config file.
-x <sym>
Trace calls to the function sym.
-s <num>
Show first num bytes of string args.
-F ltrace.conf
int mysql_real_query(addr,string,ulong);
void garbage_collect(void);
int memcached_set(addr,string,ulong,string,ulong);

ltrace -x garbage_collect
19:08:06.436926 garbage_collect() = <void> <0.221679>
19:08:15.329311 garbage_collect() = <void> <0.187546>
19:08:17.662149 garbage_collect() = <void> <0.199200>
19:08:20.486655 garbage_collect() = <void> <0.205864>
19:08:25.102302 garbage_collect() = <void> <0.214295>
19:08:35.552337 garbage_collect() = <void> <0.189172>
ltrace -x mysql_real_query
19:09:11.493395 mysql_real_query(0x19c7a500, "SELECT * FROM `users`", 21) = 0 <1.206506>
19:09:16.630981 mysql_real_query(0x1c9e0500, "SET NAMES 'UTF8'", 16) = 0 <0.000324>
19:09:16.631446 mysql_real_query(0x1c9e0500, "SET SQL_AUTO_IS_NULL=0", 22) = 0 <0.000322>
19:09:16.654231 mysql_real_query(0x1c9e0500, "COMMIT", 6) = 0 <0.000181>
ltrace -x memcached_set
memcached_set(0x15d46b80, "Status:5456561633", 21, "004bo:01{", 366) = 0 <0.001116>
memcached_set(0x15d46b80, "Status:5453277696", 21, "004bo:01{", 333) = 0 <0.000224>
memcached_set(0x15d46b80, "Status:5435377757", 21, "004bo:01{", 298) = 0 <0.001850>
memcached_set(0x15d46b80, "Status:5435122010", 21, "004bo:01{", 302) = 0 <0.000530>
memcached_set(0x15d46b80, "Status:5407037167", 21, "004bo:01{", 318) = 0 <0.000291>
memcached_set(0x15d46b80, "Status:5405690802", 21, "004bo:01{", 299) = 0 <0.000658>
memcached_set(0x15d46b80, "Status:5343957534", 21, "004bo:01{", 264) = 0 <0.000243>

gdb
the GNU debugger
gdb <program>
gdb <program> <pid>
Be sure to build with:
-ggdb
-O0

gdb walkthrough
% gdb ./test-it start gdb
(gdb) b average set breakpoint on function named average
Breakpoint 1 at 0x1f8e: file test-it.c, line 3.
(gdb) run run program
Starting program: /Users/joe/test-it
Reading symbols for shared libraries ++. done
Breakpoint 1, average (x=5, y=6) at test-it.c:3 hit breakpoint!
3 int sum = x + y;
(gdb) bt show backtrace
#0 average (x=5, y=6) at test-it.c:3
function stack
#1 0x00001fec in main () at test-it.c:12
(gdb) s
4 double avg = sum / 2.0; single step
(gdb) s
5 return avg;
(gdb) p avg
$1 = 5.5 print variables
(gdb) p sum
$2 = 11

Ruby VM stack traces
(gdb) where
#0 0x0002a55e in rb_call (klass=1386800, recv=5056455, mid=42, argc=1, argv=0xbfffe5c0,
scope=0, self=1403220) at eval.c:6125
#1 0x000226ef in rb_eval (self=1403220, n=0x1461e4) at eval.c:3493
#2 0x00026d01 in rb_yield_0 (val=5056455, self=1403220, klass=0, flags=0, avalue=0) at
eval.c:5083
#3 0x000270e8 in rb_yield (val=5056455) at eval.c:5168
#4 0x0005c30c in int_dotimes (num=1000000001) at numeric.c:2946
#5 0x00029be3 in call_cfunc (func=0x5c2a0 <int_dotimes>, recv=1000000001, len=0, argc=0,
argv=0x0) at eval.c:5759
#6 0x00028fd4 in rb_call0 (klass=1387580, recv=1000000001, id=5785, oid=5785, argc=0,
argv=0x0, body=0x152b24, flags=0) at eval.c:5911
#7 0x0002a7a7 in rb_call (klass=1387580, recv=1000000001, mid=5785, argc=0, argv=0x0,
scope=0, self=1403220) at eval.c:6158
#8 0x000226ef in rb_eval (self=1403220, n=0x146284) at eval.c:3493
#9 0x000213e3 in rb_eval (self=1403220, n=0x1461a8) at eval.c:3223
#10 0x0001ceea in eval_node (self=1403220, node=0x1461a8) at eval.c:1437
#11 0x0001d60f in ruby_exec_internal () at eval.c:1642
#12 0x0001d660 in ruby_exec () at eval.c:1662
#13 0x0001d68e in ruby_run () at eval.c:1672
#14 0x000023dc in main (argc=2, argv=0xbffff7c4, envp=0xbffff7d0) at main.c:48
rb_eval recursively executes ruby code in 1.8

Debugging Ruby Segfaults
test_segv.rb:4: [BUG] Segmentation fault
ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0]
def test
#include "ruby.h" require 'segv'
4.times do
VALUE Dir.chdir '/tmp' do
segv() Hash.new{ segv }[0]
{ end
VALUE array[1]; end
array[1000000] = NULL; end
return Qnil;
} sleep 10
test()
void
Init_segv()
{
rb_define_method(rb_cObject, "segv", segv, 0);
}

1. Attach to running process
$ ps aux | grep ruby
joe 23611 0.0 0.1 25424 7540 S Dec01 0:00 ruby test_segv.rb
$ sudo gdb ruby 23611
Attaching to program: ruby, process 23611
0x00007fa5113c0c93 in nanosleep () from /lib/libc.so.6
(gdb) c
Continuing.
Program received signal SIGBUS, Bus error.
segv () at segv.c:7
7 array[1000000] = NULL;
2. Use a coredump
Process.setrlimit Process::RLIMIT_CORE, 300*1024*1024
$ sudo mkdir /cores
$ sudo chmod 777 /cores
$ sudo sysctl kernel.core_pattern=/cores/%e.core.%s.%p.%t
$ sudo gdb ruby /cores/ruby.core.6.23611.1259781224

def test
require 'segv'
4.times do
Dir.chdir '/tmp' do
Hash.new{ segv }[0]
end
end (gdb) where
end #0 segv () at segv.c:7
#1 0x000000000041f2be in call_cfunc () at eval.c:5727
test() ...
#13 0x000000000043ba8c in rb_hash_default () at hash.c:521
...
#19 0x000000000043b92a in rb_hash_aref () at hash.c:429
...
#26 0x00000000004bb7bc in chdir_yield () at dir.c:728
#27 0x000000000041d8d7 in rb_ensure () at eval.c:5528
#28 0x00000000004bb93a in dir_s_chdir () at dir.c:816
...
#35 0x000000000041c444 in rb_yield () at eval.c:5142
#36 0x0000000000450690 in int_dotimes () at numeric.c:2834
...
#48 0x0000000000412a90 in ruby_run () at eval.c:1678
#49 0x000000000041014e in main () at main.c:48

Enough C!
What about Ruby?

perftools.rb
google-perftools for ruby
gem install perftools.rb
export RUBYOPT=”-r`gem which perftools | tail -1`”
CPUPROFILE=/tmp/myprof ruby myapp.rb
pprof.rb /tmp/myprof
http://github.com/tmm1/perftools.rb

require 'sinatra'
$ ab -c 1 -n 50 http://127.0.0.1:4567/compute
$ ab -c 1 -n 50 http://127.0.0.1:4567/sleep
get '/sleep' do
sleep 0.25 Sampling profiler:
'done'
end 232 samples total
get '/compute' do 83 samples were in /compute
proc{ |n|
a,b=0,1 118 samples had /compute on
n.times{ a,b = b,a+b } the stack but were in
b another function
}.call(10_000)
'done' /compute accounts for 50%
of process, but only 35% of
end time was in /compute itself
== Sinatra has ended his set (crowd applauds)
PROFILE: interrupts/evictions/bytes = 232/0/2152
Total: 232 samples
83 35.8% 35.8% 118 50.9% Sinatra::Application#GET /compute
56 24.1% 59.9% 56 24.1% garbage_collector
35 15.1% 75.0% 113 48.7% Integer#times

CPUPROFILE_REALTIME=1
CPUPROFILE=app.prof
CPUPROFILE=app-rt.prof

redis-rb bottleneck

why is rubygems slow?

gdb.rb
gdb with MRI hooks
gdb.rb <pid>
http://github.com/tmm1/gdb.rb

def test
require 'segv'
4.times do
Dir.chdir '/tmp' do
Hash.new{ segv }[0]
end
end
end
(gdb) ruby threads
test()
0xa3e000 main curr thread THREAD_RUNNABLE WAIT_NONE
node_vcall segv in test_segv.rb:5
node_call test in test_segv.rb:5
node_call call in test_segv.rb:5
node_call default in test_segv.rb:5
node_call [] in test_segv.rb:5
node_call test in test_segv.rb:4
node_call chdir in test_segv.rb:4
node_call test in test_segv.rb:3
node_call times in test_segv.rb:3
node_vcall test in test_segv.rb:9

(gdb) ruby threads list
0x15890 main thread THREAD_STOPPED WAIT_JOIN(0x19ef400) 4417 bytes
0x19ef4 thread THREAD_STOPPED WAIT_TIME(57.10) 6267 bytes
0x19e34 thread THREAD_STOPPED WAIT_FD(5) 10405 bytes
0x19dc4 thread THREAD_STOPPED WAIT_NONE 14237 bytes
0x19dc8 thread THREAD_STOPPED WAIT_NONE 14237 bytes
0x19dcc thread THREAD_STOPPED WAIT_NONE 14237 bytes
0x22668 thread THREAD_STOPPED WAIT_NONE 14237 bytes
0x1d630 curr thread THREAD_RUNNABLE WAIT_NONE
(gdb) ruby eval 1+2 (gdb) ruby objects
3 HEAPS 8
(gdb) ruby eval Thread.current SLOTS 1686252
#<Thread:0x1d630 run> LIVE 893327 (52.98%)
(gdb) ruby eval Thread.list.size FREE 792925 (47.02%)
8
scope 1641 (0.18%)
(gdb) ruby objects strings regexp 2255 (0.25%)
140 u'lib' data 3539 (0.40%)
158 u'0' class 3680 (0.41%)
294 u'n' hash 6196 (0.69%)
619 u'' object 8785 (0.98%)
array 13850 (1.55%)
30503 unique strings string 105350 (11.79%)
3187435 bytes node 742346 (83.10%)

rails_warden leak
(gdb) ruby objects classes
1197 MIME::Type
2657 NewRelic::MetricSpec
2719 TZInfo::TimezoneTransitionInfo
4124 Warden::Manager
4124 MethodOverrideForAll
4124 AccountMiddleware
4124 Rack::Cookies
4125 ActiveRecord::ConnectionAdapters::ConnectionManagement
4125 ActionController::Session::CookieStore
4125 ActionController::Failsafe
4125 ActionController::ParamsParser
4125 Rack::Lock
4125 ActionController::Dispatcher
4125 ActiveRecord::QueryCache
4125 ActiveSupport::MessageVerifier
4125 Rack::Head
middleware chain leaking per request

mongrel sleeper thread
0x16814c00 thread THREAD_STOPPED WAIT_TIME(0.47) 1522 bytes
node_fcall sleep in lib/mongrel/configurator.rb:285
node_fcall run in lib/mongrel/configurator.rb:285
node_fcall loop in lib/mongrel/configurator.rb:285
node_call run in lib/mongrel/configurator.rb:285
node_call initialize in lib/mongrel/configurator.rb:285
node_call new in lib/mongrel/configurator.rb:285
node_call run in bin/mongrel_rails:128
node_call run in lib/mongrel/command.rb:212
node_call run in bin/mongrel_rails:281
node_fcall (unknown) in bin/mongrel_rails:19
def run
@listeners.each {|name,s|
s.run
}
$mongrel_sleeper_thread = Thread.new { loop { sleep 1 } }
end

god memory leaks
(gdb) ruby objects arrays
elements instances
43 God::Process
94310 3
43 God::Watch
94311 3
43 God::Driver
94314 2
43 God::DriverEventQueue
94316 1
43 God::Conditions::MemoryUsage
43 God::Conditions::ProcessRunning
5369 arrays
43 God::Behaviors::CleanPidFile
2863364 member elements
45 Process::Status
86 God::Metric
many arrays with 327 God::System::SlashProcPoller
90k+ elements! 327 God::System::Process
406 God::DriverEvent
5 separate god leaks fixed by Eric
Lindvall with the help of gdb.rb!

ruby method cache
(gdb) ruby methodcache
Module#extend wipes the UnboundMethod#arity
Hash#[]=
entire method cache! Class#private
Array#freeze
(gdb) b rb_clear_cache Module#Integer
Breakpoint 1 at 0x41067b: file eval.c, line 351. Fixnum#<
(gdb) c Class#is_a?
Continuing. Fixnum#+
Class#protected
Breakpoint 1, rb_clear_cache () at eval.c:351 Class#>=
351 if (!ruby_running) return;
(gdb) ruby threads 2028 empty slots (99.02%)
0x1623000 main curr thread THREAD_RUNNABLE WAIT_NONE
node_call extend_object in sin.rb:23
node_call extend in sin.rb:23
node_call GET /other in lib/sinatra/base.rb:779
node_call GET /other in lib/sinatra/base.rb:779
node_call call in lib/sinatra/base.rb:779
node_fcall route in lib/sinatra/base.rb:474

bleak_house
ruby memory leak detector
gem install bleak_house
export RUBYOPT=”-r`gem which bleak_house | tail -1`”
ruby-bleak-house myapp.rb
bleak /tmp/bleak.5979.000.dump
http://github.com/fauna/bleak_house

191691 total objects
Final heap size 191691 filled, 220961 free
Displaying top 20 most common line/class pairs
89513 __null__:__null__:__node__
41438 __null__:__null__:String
2348 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:Array
1508 lib/ruby/gems/1.8/specifications/gettext-1.9.gemspec:14:String
1021 lib/ruby/gems/1.8/specifications/heel-0.2.0.gemspec:14:String
951 lib/ruby/site_ruby/1.8/rubygems/version.rb:111:String
935 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:String
834 lib/ruby/site_ruby/1.8/rubygems/version.rb:146:Array
BleakHouse
installs a patched version of ruby: ruby-bleak-
house
unlike gdb.rb, see where objects were created
(file:line)
create multiple dumps over time with `kill -USR2
<pid>` and compare to find leaks

Coming soon:
bleak_house++
no patches or need to recompile
ruby
“assembly metaprogramming” to
setup a trampoline on rb_new_obj
http://timetobleed.com/rewrite-
your-ruby-vm-at-runtime-to-hot-
patch-useful-features/
memory profiler
coredump a production ruby process
load the core to generate profiles
of memory usage and leaks

Questions?
@joedamato @tmm1
timetobleed.com github.com/tmm1
Thanks for listening!