Debugging Ruby
Upcoming SlideShare
Loading in...5
×
 

Debugging Ruby

on

  • 14,551 views

Debugging Ruby: Understanding and Troubleshooting the VM and your Application

Debugging Ruby: Understanding and Troubleshooting the VM and your Application

Statistics

Views

Total Views
14,551
Views on SlideShare
14,477
Embed Views
74

Actions

Likes
29
Downloads
179
Comments
2

9 Embeds 74

http://www.slideshare.net 56
http://blog.darkhax.com 5
http://verboselogging.com 4
http://www.powercram.com 3
https://twitter.com 2
http://webcache.googleusercontent.com 1
http://paper.li 1
https://si0.twimg.com 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • good
    Are you sure you want to
    Your message goes here
    Processing…
  • Congrats Aman,

    This presentation helps me a lot!

    I had worked before with gdb and valgrind, but with ruby was my first time. ;)
    Thanks for sharing that!!

    Best regards,
    Vinícius Schmidt
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Debugging Ruby Debugging Ruby Presentation Transcript

  • Debugging Ruby troubleshooting the VM and your app Aman Gupta
  • About Aman Gupta San Francisco, CA Ruby Performance Consulting (Twitter, Github, Heroku, ZenDesk) Ruby Hero 2009 EventMachine, amqp, REE, sinbook, perftools.rb, gdb.rb github.com/tmm1 @tmm1
  • The Ruby VM MRI- Ruby 1.8.{6,7} REE- Ruby 1.8.7 + patches Tools for C code Tested primarily on: Linux operating system Intel CPUs (i686 and x86_64)
  • lsof list open files lsof -nPp <pid>
  • lsof -nPp <pid> -n Inhibits the conversion of network numbers to host names for network files. -P Inhibits the conversion of port numbers to port names for network files FD TYPE SIZE NODE NAME json cwd DIR 4096 10437557 /var/www/myapp memcached rtd DIR 4096 2 / mysql txt REG 1925210 1089684 /usr/bin/ruby mem REG 48188 7061523 /json-1.1.9/ext/json/ext/generator.so http mem REG 46970 7061524 /json-1.1.9/ext/json/ext/parser.so mem REG 3428025 1131339 /memcached-0.17.4/lib/rlibmemcached.so mem REG 152972 5303443 /mysql-2.8.1/lib/mysql_api.so mem REG 154013 1089708 /usr/lib/libtcmalloc_minimal.so.0.0.0 mem REG 119288 11616294 /lib/ld-2.7.so 0u CHR 303 /dev/null 1w REG 4746405529 1106245 /usr/local/nginx/logs/error.log 2w REG 4746405529 1106245 /usr/local/nginx/logs/error.log 3u IPv4 TCP 10.8.85.66:33326->10.8.85.68:3306 (ESTABLISHED) 10u IPv4 TCP 10.8.85.66:33327->10.8.85.68:3306 (ESTABLISHED) 11u IPv4 TCP 127.0.0.1:58273->127.0.0.1:11211 (ESTABLISHED) 12u REG 20182 12463107 /tmp/RackMultipart.28957.0 33u IPv4 TCP 174.36.83.42:37466->69.63.180.21:80 (ESTABLISHED)
  • strace trace system calls and signals strace -cp <pid> strace -ttTp <pid> -o <file>
  • strace -cp <pid> -c Count time, calls, and errors for each system call and report a summary on program exit. -p pid Attach to the process with the process ID pid and begin tracing. % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 50.39 0.000064 0 1197 592 read 34.65 0.000044 0 609 writev 14.96 0.000019 0 1226 epoll_ctl 0.00 0.000000 0 4 close 0.00 0.000000 0 1 select 0.00 0.000000 0 4 socket 0.00 0.000000 0 4 4 connect 0.00 0.000000 0 1057 epoll_wait ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000127 4134 596 total
  • strace -ttTp <pid> -o <file> -t Prefix each line of the trace with the time of day. -tt If given twice, the time printed will include the microseconds. -T Show the time spent in system calls. This records the time difference between the beginning and the end of each system call. -o filename Write the trace output to the file filename rather than to stderr. 01:09:11.266949 epoll_wait(9, {{EPOLLIN, {u32=68841296, u64=68841296}}}, 4096, 50) = 1 <0.033109> 01:09:11.300102 accept(10, {sa_family=AF_INET, sin_port=38313, sin_addr="127.0.0.1"}, [1226]) = 22 <0.000014> 01:09:11.300190 fcntl(22, F_GETFL) = 0x2 (flags O_RDWR) <0.000007> 01:09:11.300237 fcntl(22, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000008> 01:09:11.300277 setsockopt(22, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008> 01:09:11.300489 accept(10, 0x7fff5d9c07d0, [1226]) = -1 EAGAIN <0.000014> 01:09:11.300547 epoll_ctl(9, EPOLL_CTL_ADD, 22, {EPOLLIN, {u32=108750368, u64=108750368}}) = 0 <0.000009> 01:09:11.300593 epoll_wait(9, {{EPOLLIN, {u32=108750368, u64=108750368}}}, 4096, 50) = 1 <0.000007> 01:09:11.300633 read(22, "GET / HTTP/1.1r"..., 16384) = 772 <0.000012> 01:09:11.301727 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000007> 01:09:11.302095 poll([{fd=5, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout) <0.000008> 01:09:11.302144 write(5, "1000000-0003SELECT * FROM `table`"..., 56) = 56 <0.000023> 01:09:11.302221 read(5, "25101,20x234m"..., 16384) = 284 <1.300897>
  • stracing ruby: SIGVTALRM 15:45:51.658164 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.658244 rt_sigreturn(0x1a) = 2207807 <0.000009> 15:45:51.678208 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.678271 rt_sigreturn(0x1a) = 0 <0.000009> 15:45:51.698161 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.698216 rt_sigreturn(0x1a) = 140734552062624 <0.000009> 15:45:51.718154 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.718192 rt_sigreturn(0x1a) = 140734552066688 <0.000009> 15:45:51.738185 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.738221 rt_sigreturn(0x1a) = 11333952 <0.000008> 15:45:51.758162 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.758216 rt_sigreturn(0x1a) = 0 <0.000009> 15:45:51.818168 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.819817 rt_sigreturn(0x1a) = 1 <0.000010> 15:45:51.838196 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- ruby 1.8 uses signals to schedule its green threads process receives a SIGVTALRM signal every 10ms
  • stracing ruby: sigprocmask % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.326334 0 3568567 rt_sigprocmask 0.00 0.000000 0 9 read 0.00 0.000000 0 10 open 0.00 0.000000 0 10 close 0.00 0.000000 0 9 fstat 0.00 0.000000 0 25 mmap ------ ----------- ----------- --------- --------- ---------------- 100.00 0.326334 3568685 0 total debian/redhat compile ruby with --enable-pthread uses a native thread timer for SIGVTALRM causes excessive calls to sigprocmask: 30% slowdown!
  • tcpdump dump traffic on a network tcpdump -i eth1 -s 1500 -nqA tcp dst port 80 tcpdump -i eth0 -s 1500 -nqA tcp dst port 3306 tcpdump -i eth1 -s 1500 -w <file> tcp dst port 80
  • tcpdump -i <eth> -s <len> -nqA <expr> tcpdump -i <eth> -w <file> <expr> -i <eth> Network interface. -s <len> Snarf len bytes of data from each packet. -n Don't convert addresses (host addresses, port numbers) to names. -q Quiet output. Print less protocol information. -A Print each packet (minus its link level header) in ASCII. -w <file> Write the raw packets to file rather than printing them out. <expr> libpcap expression, for example: tcp src port 80 tcp dst port 3306
  • tcp dst port 80 19:52:20.216294 IP 24.203.197.27.40105 > 174.37.48.236.80: tcp 438 E...*.@.l.%&.....%0....POx..%s.oP.......GET /poll_images/cld99erh0/logo.png HTTP/1.1 Accept: */* Referer: http://apps.facebook.com/realpolls/?_fb_q=1 tcp dst port 3306 19:51:06.501632 IP 10.8.85.66.50443 > 10.8.85.68.3306: tcp 98 E..."K@.@.Yy .UB .UD.....z....L............ GZ.y3b..[......W....SELECT * FROM `votes` WHERE (`poll_id` = 72621) LIMIT 1 tcpdump -w <file>
  • google-perftools Google’s CPU profiler export LD_PRELOAD=libprofiler.so export DYLD_INSERT_LIBRARIES=libprofiler.dylib CPUPROFILE=/tmp/myprof ./myapp pprof ./myapp /tmp/myprof
  • wget http://google-perftools.googlecode.com/files/google- perftools-1.4.tar.gz download tar zxvf google-perftools-1.4.tar.gz cd google-perftools-1.4 ./configure --prefix=/opt make compile sudo make install # for linux export LD_PRELOAD=/opt/lib/libprofiler.so setup # for osx export DYLD_INSERT_LIBRARIES=/opt/lib/libprofiler.dylib CPUPROFILE=/tmp/ruby.prof ruby -e' profile 5_000_000.times{ "hello world" } ' pprof `which ruby` --text /tmp/ruby.prof report
  • pprof ruby pprof ruby ruby.prof --text ruby.prof --gif Total: 103 samples 20 19.4% 19.4% 95 92.2% rb_yield_0 11 10.7% 30.1% 103 100.0% rb_eval 8 7.8% 37.9% 12 11.7% gc_sweep 3 2.9% 68.9% 52 50.5% rb_str_new3 3 2.9% 74.8% 3 2.9% obj_free 3 2.9% 77.7% 103 100.0% int_dotimes 3 2.9% 80.6% 12 11.7% gc_mark
  • Profiling MRI 10% of production VM time spent in rb_str_sub_bang String#sub! called from Time.parse return unless str.sub!(/A(d{1,2})/, '') return unless str.sub!(/A( d|d{1,2})/, '') return unless str.sub!(/A( d|d{1,2})/, '') return unless str.sub!(/A(d{1,3})/, '') return unless str.sub!(/A(d{1,2})/, '') return unless str.sub!(/A(d{1,2})/, '') switch to third_base gem ThirdBase: Fast and Easy Date/DateTime class for Ruby
  • Profiling EM + threads Total: 3763 samples 2764 73.5% catch_timer 989 26.3% memcpy 3 0.1% st_lookup 2 0.1% rb_thread_schedule 1 0.0% rb_eval 1 0.0% rb_newobj 1 0.0% rb_gc_force_recycle known issue: EM+threads = slow memcpy?? thread context switches copy the stack w/ memcpy EM allocates huge buffer on the stack solution: move buffer to the heap
  • ltrace trace library calls ltrace -cp <pid> ltrace -ttTp <pid> -o <file>
  • ltrace -c ruby threaded_em.rb % time seconds usecs/call calls function ------ ----------- ----------- --------- -------------------- 48.65 11.741295 617 19009 memcpy 30.16 7.279634 831 8751 longjmp 9.78 2.359889 135 17357 _setjmp 8.91 2.150565 285 7540 malloc 1.10 0.265946 20 13021 memset 0.81 0.195272 19 10105 __ctype_b_loc 0.35 0.084575 19 4361 strcmp 0.19 0.046163 19 2377 strlen 0.03 0.006272 23 265 realloc ------ ----------- ----------- --------- -------------------- 100.00 24.134999 82999 total ltrace -ttT -e memcpy ruby threaded_em.rb 01:24:48.769408 --- SIGVTALRM (Virtual timer expired) --- 01:24:48.769616 memcpy(0x1216000, "", 1086328) = 0x1216000 <0.000578> 01:24:48.770555 memcpy(0x6e32670, "240&343v", 1086328) = 0x6e32670 <0.000418> 01:24:49.899414 --- SIGVTALRM (Virtual timer expired) --- 01:24:49.899490 memcpy(0x1320000, "", 1082584) = 0x1320000 <0.000628> 01:24:49.900474 memcpy(0x6e32670, "", 1086328) = 0x6e32670 <0.000479>
  • ltrace/libdl trace dlopen’d library calls ltrace -F <conf> -bg -x <symbol> -p <pid> http://github.com/ice799/ltrace/tree/libdl
  • ltrace -F <conf> -b -g -x <sym> -b Ignore signals. -g Ignore libraries linked at compile time. -F <conf> Read prototypes from config file. -x <sym> Trace calls to the function sym. -s <num> Show first num bytes of string args. -F ltrace.conf int mysql_real_query(addr,string,ulong); void garbage_collect(void); int memcached_set(addr,string,ulong,string,ulong);
  • ltrace -x garbage_collect 19:08:06.436926 garbage_collect() = <void> <0.221679> 19:08:15.329311 garbage_collect() = <void> <0.187546> 19:08:17.662149 garbage_collect() = <void> <0.199200> 19:08:20.486655 garbage_collect() = <void> <0.205864> 19:08:25.102302 garbage_collect() = <void> <0.214295> 19:08:35.552337 garbage_collect() = <void> <0.189172> ltrace -x mysql_real_query 19:09:11.493395 mysql_real_query(0x19c7a500, "SELECT * FROM `users`", 21) = 0 <1.206506> 19:09:16.630981 mysql_real_query(0x1c9e0500, "SET NAMES 'UTF8'", 16) = 0 <0.000324> 19:09:16.631446 mysql_real_query(0x1c9e0500, "SET SQL_AUTO_IS_NULL=0", 22) = 0 <0.000322> 19:09:16.654231 mysql_real_query(0x1c9e0500, "COMMIT", 6) = 0 <0.000181> ltrace -x memcached_set memcached_set(0x15d46b80, "Status:5456561633", 21, "004bo:01{", 366) = 0 <0.001116> memcached_set(0x15d46b80, "Status:5453277696", 21, "004bo:01{", 333) = 0 <0.000224> memcached_set(0x15d46b80, "Status:5435377757", 21, "004bo:01{", 298) = 0 <0.001850> memcached_set(0x15d46b80, "Status:5435122010", 21, "004bo:01{", 302) = 0 <0.000530> memcached_set(0x15d46b80, "Status:5407037167", 21, "004bo:01{", 318) = 0 <0.000291> memcached_set(0x15d46b80, "Status:5405690802", 21, "004bo:01{", 299) = 0 <0.000658> memcached_set(0x15d46b80, "Status:5343957534", 21, "004bo:01{", 264) = 0 <0.000243>
  • gdb the GNU debugger gdb <program> gdb <program> <pid> Be sure to build with: -ggdb -O0
  • gdb walkthrough % gdb ./test-it start gdb (gdb) b average set breakpoint on function named average Breakpoint 1 at 0x1f8e: file test-it.c, line 3. (gdb) run run program Starting program: /Users/joe/test-it Reading symbols for shared libraries ++. done Breakpoint 1, average (x=5, y=6) at test-it.c:3 hit breakpoint! 3 int sum = x + y; (gdb) bt show backtrace #0 average (x=5, y=6) at test-it.c:3 function stack #1 0x00001fec in main () at test-it.c:12 (gdb) s 4 double avg = sum / 2.0; single step (gdb) s 5 return avg; (gdb) p avg $1 = 5.5 print variables (gdb) p sum $2 = 11
  • Ruby VM stack traces (gdb) where #0 0x0002a55e in rb_call (klass=1386800, recv=5056455, mid=42, argc=1, argv=0xbfffe5c0, scope=0, self=1403220) at eval.c:6125 #1 0x000226ef in rb_eval (self=1403220, n=0x1461e4) at eval.c:3493 #2 0x00026d01 in rb_yield_0 (val=5056455, self=1403220, klass=0, flags=0, avalue=0) at eval.c:5083 #3 0x000270e8 in rb_yield (val=5056455) at eval.c:5168 #4 0x0005c30c in int_dotimes (num=1000000001) at numeric.c:2946 #5 0x00029be3 in call_cfunc (func=0x5c2a0 <int_dotimes>, recv=1000000001, len=0, argc=0, argv=0x0) at eval.c:5759 #6 0x00028fd4 in rb_call0 (klass=1387580, recv=1000000001, id=5785, oid=5785, argc=0, argv=0x0, body=0x152b24, flags=0) at eval.c:5911 #7 0x0002a7a7 in rb_call (klass=1387580, recv=1000000001, mid=5785, argc=0, argv=0x0, scope=0, self=1403220) at eval.c:6158 #8 0x000226ef in rb_eval (self=1403220, n=0x146284) at eval.c:3493 #9 0x000213e3 in rb_eval (self=1403220, n=0x1461a8) at eval.c:3223 #10 0x0001ceea in eval_node (self=1403220, node=0x1461a8) at eval.c:1437 #11 0x0001d60f in ruby_exec_internal () at eval.c:1642 #12 0x0001d660 in ruby_exec () at eval.c:1662 #13 0x0001d68e in ruby_run () at eval.c:1672 #14 0x000023dc in main (argc=2, argv=0xbffff7c4, envp=0xbffff7d0) at main.c:48 rb_eval recursively executes ruby code in 1.8
  • Debugging Ruby Segfaults test_segv.rb:4: [BUG] Segmentation fault ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0] def test #include "ruby.h" require 'segv' 4.times do VALUE Dir.chdir '/tmp' do segv() Hash.new{ segv }[0] { end VALUE array[1]; end array[1000000] = NULL; end return Qnil; } sleep 10 test() void Init_segv() { rb_define_method(rb_cObject, "segv", segv, 0); }
  • 1. Attach to running process $ ps aux | grep ruby joe 23611 0.0 0.1 25424 7540 S Dec01 0:00 ruby test_segv.rb $ sudo gdb ruby 23611 Attaching to program: ruby, process 23611 0x00007fa5113c0c93 in nanosleep () from /lib/libc.so.6 (gdb) c Continuing. Program received signal SIGBUS, Bus error. segv () at segv.c:7 7 array[1000000] = NULL; 2. Use a coredump Process.setrlimit Process::RLIMIT_CORE, 300*1024*1024 $ sudo mkdir /cores $ sudo chmod 777 /cores $ sudo sysctl kernel.core_pattern=/cores/%e.core.%s.%p.%t $ sudo gdb ruby /cores/ruby.core.6.23611.1259781224
  • def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end end (gdb) where end #0 segv () at segv.c:7 #1 0x000000000041f2be in call_cfunc () at eval.c:5727 test() ... #13 0x000000000043ba8c in rb_hash_default () at hash.c:521 ... #19 0x000000000043b92a in rb_hash_aref () at hash.c:429 ... #26 0x00000000004bb7bc in chdir_yield () at dir.c:728 #27 0x000000000041d8d7 in rb_ensure () at eval.c:5528 #28 0x00000000004bb93a in dir_s_chdir () at dir.c:816 ... #35 0x000000000041c444 in rb_yield () at eval.c:5142 #36 0x0000000000450690 in int_dotimes () at numeric.c:2834 ... #48 0x0000000000412a90 in ruby_run () at eval.c:1678 #49 0x000000000041014e in main () at main.c:48
  • Enough C! What about Ruby?
  • perftools.rb google-perftools for ruby gem install perftools.rb export RUBYOPT=”-r`gem which perftools | tail -1`” CPUPROFILE=/tmp/myprof ruby myapp.rb pprof.rb /tmp/myprof http://github.com/tmm1/perftools.rb
  • require 'sinatra' $ ab -c 1 -n 50 http://127.0.0.1:4567/compute $ ab -c 1 -n 50 http://127.0.0.1:4567/sleep get '/sleep' do sleep 0.25 Sampling profiler: 'done' end 232 samples total get '/compute' do 83 samples were in /compute proc{ |n| a,b=0,1 118 samples had /compute on n.times{ a,b = b,a+b } the stack but were in b another function }.call(10_000) 'done' /compute accounts for 50% of process, but only 35% of end time was in /compute itself == Sinatra has ended his set (crowd applauds) PROFILE: interrupts/evictions/bytes = 232/0/2152 Total: 232 samples 83 35.8% 35.8% 118 50.9% Sinatra::Application#GET /compute 56 24.1% 59.9% 56 24.1% garbage_collector 35 15.1% 75.0% 113 48.7% Integer#times
  • CPUPROFILE_REALTIME=1 CPUPROFILE=app.prof CPUPROFILE=app-rt.prof
  • redis-rb bottleneck
  • why is rubygems slow?
  • gdb.rb gdb with MRI hooks gdb.rb <pid> http://github.com/tmm1/gdb.rb
  • def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end end end (gdb) ruby threads test() 0xa3e000 main curr thread THREAD_RUNNABLE WAIT_NONE node_vcall segv in test_segv.rb:5 node_call test in test_segv.rb:5 node_call call in test_segv.rb:5 node_call default in test_segv.rb:5 node_call [] in test_segv.rb:5 node_call test in test_segv.rb:4 node_call chdir in test_segv.rb:4 node_call test in test_segv.rb:3 node_call times in test_segv.rb:3 node_vcall test in test_segv.rb:9
  • (gdb) ruby threads list 0x15890 main thread THREAD_STOPPED WAIT_JOIN(0x19ef400) 4417 bytes 0x19ef4 thread THREAD_STOPPED WAIT_TIME(57.10) 6267 bytes 0x19e34 thread THREAD_STOPPED WAIT_FD(5) 10405 bytes 0x19dc4 thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x19dc8 thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x19dcc thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x22668 thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x1d630 curr thread THREAD_RUNNABLE WAIT_NONE (gdb) ruby eval 1+2 (gdb) ruby objects 3 HEAPS 8 (gdb) ruby eval Thread.current SLOTS 1686252 #<Thread:0x1d630 run> LIVE 893327 (52.98%) (gdb) ruby eval Thread.list.size FREE 792925 (47.02%) 8 scope 1641 (0.18%) (gdb) ruby objects strings regexp 2255 (0.25%) 140 u'lib' data 3539 (0.40%) 158 u'0' class 3680 (0.41%) 294 u'n' hash 6196 (0.69%) 619 u'' object 8785 (0.98%) array 13850 (1.55%) 30503 unique strings string 105350 (11.79%) 3187435 bytes node 742346 (83.10%)
  • rails_warden leak (gdb) ruby objects classes 1197 MIME::Type 2657 NewRelic::MetricSpec 2719 TZInfo::TimezoneTransitionInfo 4124 Warden::Manager 4124 MethodOverrideForAll 4124 AccountMiddleware 4124 Rack::Cookies 4125 ActiveRecord::ConnectionAdapters::ConnectionManagement 4125 ActionController::Session::CookieStore 4125 ActionController::Failsafe 4125 ActionController::ParamsParser 4125 Rack::Lock 4125 ActionController::Dispatcher 4125 ActiveRecord::QueryCache 4125 ActiveSupport::MessageVerifier 4125 Rack::Head middleware chain leaking per request
  • mongrel sleeper thread 0x16814c00 thread THREAD_STOPPED WAIT_TIME(0.47) 1522 bytes node_fcall sleep in lib/mongrel/configurator.rb:285 node_fcall run in lib/mongrel/configurator.rb:285 node_fcall loop in lib/mongrel/configurator.rb:285 node_call run in lib/mongrel/configurator.rb:285 node_call initialize in lib/mongrel/configurator.rb:285 node_call new in lib/mongrel/configurator.rb:285 node_call run in bin/mongrel_rails:128 node_call run in lib/mongrel/command.rb:212 node_call run in bin/mongrel_rails:281 node_fcall (unknown) in bin/mongrel_rails:19 def run @listeners.each {|name,s| s.run } $mongrel_sleeper_thread = Thread.new { loop { sleep 1 } } end
  • god memory leaks (gdb) ruby objects arrays elements instances 43 God::Process 94310 3 43 God::Watch 94311 3 43 God::Driver 94314 2 43 God::DriverEventQueue 94316 1 43 God::Conditions::MemoryUsage 43 God::Conditions::ProcessRunning 5369 arrays 43 God::Behaviors::CleanPidFile 2863364 member elements 45 Process::Status 86 God::Metric many arrays with 327 God::System::SlashProcPoller 90k+ elements! 327 God::System::Process 406 God::DriverEvent 5 separate god leaks fixed by Eric Lindvall with the help of gdb.rb!
  • ruby method cache (gdb) ruby methodcache Module#extend wipes the UnboundMethod#arity Hash#[]= entire method cache! Class#private Array#freeze (gdb) b rb_clear_cache Module#Integer Breakpoint 1 at 0x41067b: file eval.c, line 351. Fixnum#< (gdb) c Class#is_a? Continuing. Fixnum#+ Class#protected Breakpoint 1, rb_clear_cache () at eval.c:351 Class#>= 351 if (!ruby_running) return; (gdb) ruby threads 2028 empty slots (99.02%) 0x1623000 main curr thread THREAD_RUNNABLE WAIT_NONE node_call extend_object in sin.rb:23 node_call extend in sin.rb:23 node_call GET /other in lib/sinatra/base.rb:779 node_call GET /other in lib/sinatra/base.rb:779 node_call call in lib/sinatra/base.rb:779 node_fcall route in lib/sinatra/base.rb:474
  • bleak_house ruby memory leak detector gem install bleak_house export RUBYOPT=”-r`gem which bleak_house | tail -1`” ruby-bleak-house myapp.rb bleak /tmp/bleak.5979.000.dump http://github.com/fauna/bleak_house
  • 191691 total objects Final heap size 191691 filled, 220961 free Displaying top 20 most common line/class pairs 89513 __null__:__null__:__node__ 41438 __null__:__null__:String 2348 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:Array 1508 lib/ruby/gems/1.8/specifications/gettext-1.9.gemspec:14:String 1021 lib/ruby/gems/1.8/specifications/heel-0.2.0.gemspec:14:String 951 lib/ruby/site_ruby/1.8/rubygems/version.rb:111:String 935 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:String 834 lib/ruby/site_ruby/1.8/rubygems/version.rb:146:Array BleakHouse installs a patched version of ruby: ruby-bleak- house unlike gdb.rb, see where objects were created (file:line) create multiple dumps over time with `kill -USR2 <pid>` and compare to find leaks
  • Coming soon: bleak_house++ no patches or need to recompile ruby “assembly metaprogramming” to setup a trampoline on rb_new_obj http://timetobleed.com/rewrite- your-ruby-vm-at-runtime-to-hot- patch-useful-features/ memory profiler coredump a production ruby process load the core to generate profiles of memory usage and leaks
  • Questions? @joedamato @tmm1 timetobleed.com github.com/tmm1 Thanks for listening!