SlideShare a Scribd company logo
1 of 46
Download to read offline
Debugging Ruby
troubleshooting the VM and your app
            Aman Gupta
About Aman Gupta
San Francisco, CA

Ruby Performance Consulting
(Twitter, Github, Heroku, ZenDesk)

Ruby Hero 2009

EventMachine, amqp, REE, sinbook,
perftools.rb, gdb.rb

github.com/tmm1

@tmm1
The Ruby VM
MRI- Ruby 1.8.{6,7}

REE- Ruby 1.8.7 + patches

Tools for C code

Tested primarily on:

 Linux operating system

 Intel CPUs (i686 and x86_64)
lsof
list open files


lsof -nPp <pid>
lsof -nPp <pid>
-n
Inhibits the conversion of network numbers to host names for
network files.

-P
Inhibits the conversion of port numbers to port names for network
files
   FD  TYPE    SIZE    NODE NAME
                                                            json
  cwd     DIR       4096   10437557   /var/www/myapp                           memcached
  rtd     DIR       4096          2   /
                                                                                  mysql
  txt     REG    1925210    1089684   /usr/bin/ruby
  mem     REG      48188    7061523   /json-1.1.9/ext/json/ext/generator.so        http
  mem     REG      46970    7061524   /json-1.1.9/ext/json/ext/parser.so
  mem     REG    3428025    1131339   /memcached-0.17.4/lib/rlibmemcached.so
  mem     REG     152972    5303443   /mysql-2.8.1/lib/mysql_api.so
  mem     REG     154013    1089708   /usr/lib/libtcmalloc_minimal.so.0.0.0
  mem     REG     119288   11616294   /lib/ld-2.7.so
    0u    CHR                   303   /dev/null
    1w    REG 4746405529    1106245   /usr/local/nginx/logs/error.log
    2w    REG 4746405529    1106245   /usr/local/nginx/logs/error.log
    3u   IPv4                   TCP   10.8.85.66:33326->10.8.85.68:3306 (ESTABLISHED)
   10u   IPv4                   TCP   10.8.85.66:33327->10.8.85.68:3306 (ESTABLISHED)
   11u   IPv4                   TCP   127.0.0.1:58273->127.0.0.1:11211 (ESTABLISHED)
   12u    REG      20182   12463107   /tmp/RackMultipart.28957.0
   33u   IPv4                   TCP   174.36.83.42:37466->69.63.180.21:80 (ESTABLISHED)
strace
trace system calls and signals


       strace -cp <pid>

 strace -ttTp <pid> -o <file>
strace -cp <pid>
-c
Count time, calls, and errors for each system call and report a
summary on program exit.

-p pid
Attach to the process with the process ID pid and begin tracing.




        % time     seconds usecs/call      calls    errors syscall
        ------ ----------- ----------- --------- --------- ----------------
         50.39    0.000064           0      1197       592 read
         34.65    0.000044           0       609           writev
         14.96    0.000019           0      1226           epoll_ctl
          0.00    0.000000           0         4           close
          0.00    0.000000           0         1           select
          0.00    0.000000           0         4           socket
          0.00    0.000000           0         4         4 connect
          0.00    0.000000           0      1057           epoll_wait
        ------ ----------- ----------- --------- --------- ----------------
        100.00    0.000127                  4134       596 total
strace -ttTp <pid> -o <file>
-t
Prefix each line of the trace with the time of day.

-tt
If given twice, the time printed will include the microseconds.

-T
Show the time spent in system calls. This records the time
difference between the beginning and the end of each system call.

-o filename
Write the trace output to the file filename rather than to stderr.

01:09:11.266949   epoll_wait(9, {{EPOLLIN, {u32=68841296, u64=68841296}}}, 4096, 50) = 1 <0.033109>
01:09:11.300102   accept(10, {sa_family=AF_INET, sin_port=38313, sin_addr="127.0.0.1"}, [1226]) = 22 <0.000014>
01:09:11.300190   fcntl(22, F_GETFL) = 0x2 (flags O_RDWR) <0.000007>
01:09:11.300237   fcntl(22, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000008>
01:09:11.300277   setsockopt(22, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>
01:09:11.300489   accept(10, 0x7fff5d9c07d0, [1226]) = -1 EAGAIN <0.000014>
01:09:11.300547   epoll_ctl(9, EPOLL_CTL_ADD, 22, {EPOLLIN, {u32=108750368, u64=108750368}}) = 0 <0.000009>
01:09:11.300593   epoll_wait(9, {{EPOLLIN, {u32=108750368, u64=108750368}}}, 4096, 50) = 1 <0.000007>
01:09:11.300633   read(22, "GET / HTTP/1.1r"..., 16384) = 772 <0.000012>
01:09:11.301727   rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000007>
01:09:11.302095   poll([{fd=5, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout) <0.000008>
01:09:11.302144   write(5, "1000000-0003SELECT * FROM `table`"..., 56) = 56 <0.000023>
01:09:11.302221   read(5, "25101,20x234m"..., 16384) = 284 <1.300897>
stracing ruby: SIGVTALRM
   15:45:51.658164   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---
   15:45:51.658244   rt_sigreturn(0x1a)        = 2207807 <0.000009>
   15:45:51.678208   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---
   15:45:51.678271   rt_sigreturn(0x1a)        = 0 <0.000009>
   15:45:51.698161   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---
   15:45:51.698216   rt_sigreturn(0x1a)        = 140734552062624 <0.000009>
   15:45:51.718154   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---
   15:45:51.718192   rt_sigreturn(0x1a)        = 140734552066688 <0.000009>
   15:45:51.738185   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---
   15:45:51.738221   rt_sigreturn(0x1a)        = 11333952 <0.000008>
   15:45:51.758162   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---
   15:45:51.758216   rt_sigreturn(0x1a)        = 0 <0.000009>
   15:45:51.818168   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---
   15:45:51.819817   rt_sigreturn(0x1a)        = 1 <0.000010>
   15:45:51.838196   --- SIGVTALRM (Virtual   timer expired) @ 0 (0) ---



ruby 1.8 uses signals to schedule its green threads

process receives a SIGVTALRM signal every 10ms
stracing ruby: sigprocmask
% time     seconds usecs/call      calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.326334           0   3568567           rt_sigprocmask
  0.00    0.000000           0         9           read
  0.00    0.000000           0        10           open
  0.00    0.000000           0        10           close
  0.00    0.000000           0         9           fstat
  0.00    0.000000           0        25           mmap
------ ----------- ----------- --------- --------- ----------------
100.00    0.326334               3568685         0 total



  debian/redhat compile ruby with --enable-pthread

  uses a native thread timer for SIGVTALRM

  causes excessive calls to sigprocmask: 30% slowdown!
tcpdump
dump traffic on a network

  tcpdump -i eth1 -s 1500 -nqA
         tcp dst port 80

  tcpdump -i eth0 -s 1500 -nqA
        tcp dst port 3306


tcpdump -i eth1 -s 1500 -w <file>
         tcp dst port 80
tcpdump -i <eth> -s <len> -nqA <expr>
  tcpdump -i <eth> -w <file> <expr>
-i <eth>
Network interface.

-s <len>
Snarf len bytes of data from each packet.

-n
Don't convert addresses (host addresses, port numbers) to names.

-q
Quiet output.   Print less protocol information.

-A
Print each packet (minus its link level header) in ASCII.

-w <file>
Write the raw packets to file rather than printing them out.

<expr>
libpcap expression, for example:
  tcp src port 80
  tcp dst port 3306
tcp dst port 80
19:52:20.216294 IP 24.203.197.27.40105 > 174.37.48.236.80: tcp 438
E...*.@.l.%&.....%0....POx..%s.oP.......GET /poll_images/cld99erh0/logo.png HTTP/1.1
Accept: */*
Referer: http://apps.facebook.com/realpolls/?_fb_q=1



                   tcp dst port 3306
   19:51:06.501632 IP 10.8.85.66.50443 > 10.8.85.68.3306: tcp 98
   E..."K@.@.Yy
   .UB
   .UD.....z....L............
   GZ.y3b..[......W....SELECT * FROM `votes` WHERE (`poll_id` = 72621) LIMIT 1



                   tcpdump -w <file>
google-perftools
       Google’s CPU profiler


       export LD_PRELOAD=libprofiler.so

export DYLD_INSERT_LIBRARIES=libprofiler.dylib


       CPUPROFILE=/tmp/myprof ./myapp


          pprof ./myapp /tmp/myprof
wget http://google-perftools.googlecode.com/files/google-
perftools-1.4.tar.gz                                 download
tar zxvf google-perftools-1.4.tar.gz
cd google-perftools-1.4

./configure --prefix=/opt
make                                                 compile
sudo make install

# for linux
export LD_PRELOAD=/opt/lib/libprofiler.so            setup

# for osx
export DYLD_INSERT_LIBRARIES=/opt/lib/libprofiler.dylib

CPUPROFILE=/tmp/ruby.prof ruby -e'                   profile
  5_000_000.times{ "hello world" }
'

pprof `which ruby` --text /tmp/ruby.prof             report
pprof ruby                                  pprof ruby
  ruby.prof --text                            ruby.prof --gif
Total: 103 samples
  20 19.4% 19.4%      95 92.2% rb_yield_0
  11 10.7% 30.1%     103 100.0% rb_eval
   8   7.8% 37.9%     12 11.7% gc_sweep
   3   2.9% 68.9%     52 50.5% rb_str_new3
   3   2.9% 74.8%      3   2.9% obj_free
   3   2.9% 77.7%    103 100.0% int_dotimes
   3   2.9% 80.6%     12 11.7% gc_mark
Profiling MRI
                      10% of production VM
                      time spent in
                      rb_str_sub_bang

                      String#sub!

                      called from Time.parse
return   unless   str.sub!(/A(d{1,2})/, '')
return   unless   str.sub!(/A( d|d{1,2})/, '')
return   unless   str.sub!(/A( d|d{1,2})/, '')
return   unless   str.sub!(/A(d{1,3})/, '')
return   unless   str.sub!(/A(d{1,2})/, '')
return   unless   str.sub!(/A(d{1,2})/, '')

                      switch to third_base gem

                      ThirdBase: Fast and Easy
                      Date/DateTime class for
                      Ruby
Profiling EM + threads
           Total: 3763 samples
            2764 73.5% catch_timer
             989 26.3% memcpy
               3   0.1% st_lookup
               2   0.1% rb_thread_schedule
               1   0.0% rb_eval
               1   0.0% rb_newobj
               1   0.0% rb_gc_force_recycle

              known issue: EM+threads =
              slow

              memcpy??

              thread context switches
              copy the stack w/ memcpy

              EM allocates huge buffer
              on the stack

              solution: move buffer to
              the heap
ltrace
    trace library calls


      ltrace -cp <pid>

ltrace -ttTp <pid> -o <file>
ltrace -c ruby threaded_em.rb
         % time     seconds usecs/call      calls       function
         ------ ----------- ----------- --------- --------------------
          48.65   11.741295         617     19009 memcpy
          30.16    7.279634         831      8751 longjmp
           9.78    2.359889         135     17357 _setjmp
           8.91    2.150565         285      7540 malloc
           1.10    0.265946          20     13021 memset
           0.81    0.195272          19     10105 __ctype_b_loc
           0.35    0.084575          19      4361 strcmp
           0.19    0.046163          19      2377 strlen
           0.03    0.006272          23       265 realloc
         ------ ----------- ----------- --------- --------------------
         100.00   24.134999                 82999 total


  ltrace -ttT -e memcpy ruby threaded_em.rb
01:24:48.769408 --- SIGVTALRM (Virtual timer expired) ---
01:24:48.769616 memcpy(0x1216000, "", 1086328)   = 0x1216000 <0.000578>
01:24:48.770555 memcpy(0x6e32670, "240&343v", 1086328) = 0x6e32670 <0.000418>

01:24:49.899414 --- SIGVTALRM (Virtual timer expired) ---
01:24:49.899490 memcpy(0x1320000, "", 1082584)   = 0x1320000 <0.000628>
01:24:49.900474 memcpy(0x6e32670, "", 1086328) = 0x6e32670 <0.000479>
ltrace/libdl
  trace dlopen’d library calls



ltrace -F <conf> -bg -x <symbol>
            -p <pid>



http://github.com/ice799/ltrace/tree/libdl
ltrace -F <conf> -b -g -x <sym>
-b
Ignore signals.

-g
Ignore libraries linked at compile time.

-F <conf>
Read prototypes from config file.

-x <sym>
Trace calls to the function sym.

-s <num>
Show first num bytes of string args.



                  -F ltrace.conf
    int mysql_real_query(addr,string,ulong);
    void garbage_collect(void);
    int memcached_set(addr,string,ulong,string,ulong);
ltrace -x garbage_collect
     19:08:06.436926   garbage_collect()   =   <void>   <0.221679>
     19:08:15.329311   garbage_collect()   =   <void>   <0.187546>
     19:08:17.662149   garbage_collect()   =   <void>   <0.199200>
     19:08:20.486655   garbage_collect()   =   <void>   <0.205864>
     19:08:25.102302   garbage_collect()   =   <void>   <0.214295>
     19:08:35.552337   garbage_collect()   =   <void>   <0.189172>



         ltrace -x mysql_real_query
19:09:11.493395   mysql_real_query(0x19c7a500,   "SELECT * FROM `users`", 21)              =   0   <1.206506>
19:09:16.630981   mysql_real_query(0x1c9e0500,   "SET NAMES 'UTF8'", 16)                   =   0   <0.000324>
19:09:16.631446   mysql_real_query(0x1c9e0500,   "SET SQL_AUTO_IS_NULL=0", 22)             =   0   <0.000322>
19:09:16.654231   mysql_real_query(0x1c9e0500,   "COMMIT", 6)                              =   0   <0.000181>



             ltrace -x memcached_set
 memcached_set(0x15d46b80,   "Status:5456561633",       21,   "004bo:01{",   366)   =   0   <0.001116>
 memcached_set(0x15d46b80,   "Status:5453277696",       21,   "004bo:01{",   333)   =   0   <0.000224>
 memcached_set(0x15d46b80,   "Status:5435377757",       21,   "004bo:01{",   298)   =   0   <0.001850>
 memcached_set(0x15d46b80,   "Status:5435122010",       21,   "004bo:01{",   302)   =   0   <0.000530>
 memcached_set(0x15d46b80,   "Status:5407037167",       21,   "004bo:01{",   318)   =   0   <0.000291>
 memcached_set(0x15d46b80,   "Status:5405690802",       21,   "004bo:01{",   299)   =   0   <0.000658>
 memcached_set(0x15d46b80,   "Status:5343957534",       21,   "004bo:01{",   264)   =   0   <0.000243>
gdb
 the GNU debugger


   gdb <program>
gdb <program> <pid>


  Be sure to build with:
           -ggdb
            -O0
gdb walkthrough
% gdb ./test-it           start gdb
(gdb) b average           set breakpoint on function named average
Breakpoint 1 at 0x1f8e: file test-it.c, line 3.

(gdb) run          run program
Starting program: /Users/joe/test-it
Reading symbols for shared libraries ++. done
Breakpoint 1, average (x=5, y=6) at test-it.c:3             hit breakpoint!
3	    int sum = x + y;

(gdb) bt          show backtrace
#0 average (x=5, y=6) at test-it.c:3
                                                       function stack
#1 0x00001fec in main () at test-it.c:12

(gdb)   s
4	      double avg = sum / 2.0;   single step
(gdb)   s
5	      return avg;

(gdb) p avg
$1 = 5.5                             print variables

(gdb) p sum
$2 = 11
Ruby VM stack traces
(gdb) where
#0 0x0002a55e in rb_call (klass=1386800, recv=5056455, mid=42, argc=1, argv=0xbfffe5c0,
scope=0, self=1403220) at eval.c:6125
#1 0x000226ef in rb_eval (self=1403220, n=0x1461e4) at eval.c:3493
#2 0x00026d01 in rb_yield_0 (val=5056455, self=1403220, klass=0, flags=0, avalue=0) at
eval.c:5083
#3 0x000270e8 in rb_yield (val=5056455) at eval.c:5168
#4 0x0005c30c in int_dotimes (num=1000000001) at numeric.c:2946
#5 0x00029be3 in call_cfunc (func=0x5c2a0 <int_dotimes>, recv=1000000001, len=0, argc=0,
argv=0x0) at eval.c:5759
#6 0x00028fd4 in rb_call0 (klass=1387580, recv=1000000001, id=5785, oid=5785, argc=0,
argv=0x0, body=0x152b24, flags=0) at eval.c:5911
#7 0x0002a7a7 in rb_call (klass=1387580, recv=1000000001, mid=5785, argc=0, argv=0x0,
scope=0, self=1403220) at eval.c:6158
#8 0x000226ef in rb_eval (self=1403220, n=0x146284) at eval.c:3493
#9 0x000213e3 in rb_eval (self=1403220, n=0x1461a8) at eval.c:3223
#10 0x0001ceea in eval_node (self=1403220, node=0x1461a8) at eval.c:1437
#11 0x0001d60f in ruby_exec_internal () at eval.c:1642
#12 0x0001d660 in ruby_exec () at eval.c:1662
#13 0x0001d68e in ruby_run () at eval.c:1672
#14 0x000023dc in main (argc=2, argv=0xbffff7c4, envp=0xbffff7d0) at main.c:48


rb_eval recursively executes ruby code in 1.8
Debugging Ruby Segfaults
 test_segv.rb:4: [BUG] Segmentation fault
 ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0]

                             def test
  #include "ruby.h"            require 'segv'
                               4.times do
  VALUE                          Dir.chdir '/tmp' do
  segv()                           Hash.new{ segv }[0]
  {                              end
    VALUE array[1];            end
    array[1000000] = NULL;   end
    return Qnil;
  }                          sleep 10
                             test()
  void
  Init_segv()
  {
    rb_define_method(rb_cObject, "segv", segv, 0);
  }
1. Attach to running process
 $ ps aux | grep ruby
 joe 23611 0.0 0.1      25424   7540 S Dec01 0:00 ruby test_segv.rb

 $ sudo gdb ruby 23611
 Attaching to program: ruby, process 23611
 0x00007fa5113c0c93 in nanosleep () from /lib/libc.so.6
 (gdb) c
 Continuing.

 Program received signal SIGBUS, Bus error.
 segv () at segv.c:7
 7	   array[1000000] = NULL;


2. Use a coredump
 Process.setrlimit Process::RLIMIT_CORE, 300*1024*1024
 $ sudo mkdir /cores
 $ sudo chmod 777 /cores
 $ sudo sysctl kernel.core_pattern=/cores/%e.core.%s.%p.%t

 $ sudo gdb ruby /cores/ruby.core.6.23611.1259781224
def test
  require 'segv'
  4.times do
    Dir.chdir '/tmp' do
       Hash.new{ segv }[0]
    end
  end     (gdb) where
end       #0 segv () at segv.c:7
          #1 0x000000000041f2be in    call_cfunc () at eval.c:5727
test()    ...
          #13 0x000000000043ba8c in   rb_hash_default () at hash.c:521
          ...
          #19 0x000000000043b92a in   rb_hash_aref () at hash.c:429
          ...
          #26 0x00000000004bb7bc in   chdir_yield () at dir.c:728
          #27 0x000000000041d8d7 in   rb_ensure () at eval.c:5528
          #28 0x00000000004bb93a in   dir_s_chdir () at dir.c:816
          ...
          #35 0x000000000041c444 in   rb_yield () at eval.c:5142
          #36 0x0000000000450690 in   int_dotimes () at numeric.c:2834
          ...
          #48 0x0000000000412a90 in   ruby_run () at eval.c:1678
          #49 0x000000000041014e in   main () at main.c:48
Enough C!

What about Ruby?
perftools.rb
      google-perftools for ruby

            gem install perftools.rb


export RUBYOPT=”-r`gem which perftools | tail -1`”

      CPUPROFILE=/tmp/myprof ruby myapp.rb

              pprof.rb /tmp/myprof



     http://github.com/tmm1/perftools.rb
require 'sinatra'
                        $ ab -c 1 -n 50 http://127.0.0.1:4567/compute
                        $ ab -c 1 -n 50 http://127.0.0.1:4567/sleep
get '/sleep' do
  sleep 0.25                          Sampling profiler:
  'done'
end                                       232 samples total

get '/compute' do                         83 samples were in /compute
  proc{ |n|
    a,b=0,1                               118 samples had /compute on
    n.times{ a,b = b,a+b }                the stack but were in
    b                                     another function
  }.call(10_000)
  'done'                                  /compute accounts for 50%
                                          of process, but only 35% of
end                                       time was in /compute itself
   == Sinatra has ended his set (crowd applauds)
   PROFILE: interrupts/evictions/bytes = 232/0/2152

   Total: 232 samples
         83 35.8% 35.8%        118   50.9% Sinatra::Application#GET /compute
         56 24.1% 59.9%         56   24.1% garbage_collector
         35 15.1% 75.0%        113   48.7% Integer#times
CPUPROFILE_REALTIME=1
                         CPUPROFILE=app.prof
CPUPROFILE=app-rt.prof
redis-rb bottleneck
why is rubygems slow?
gdb.rb
  gdb with MRI hooks



      gdb.rb <pid>




http://github.com/tmm1/gdb.rb
def test
  require 'segv'
  4.times do
    Dir.chdir '/tmp' do
       Hash.new{ segv }[0]
    end
  end
end
            (gdb) ruby threads
test()
             0xa3e000 main curr thread THREAD_RUNNABLE WAIT_NONE
                   node_vcall     segv in test_segv.rb:5
                   node_call      test in test_segv.rb:5
                   node_call      call in test_segv.rb:5
                   node_call      default in test_segv.rb:5
                   node_call      [] in test_segv.rb:5
                   node_call      test in test_segv.rb:4
                   node_call      chdir in test_segv.rb:4
                   node_call      test in test_segv.rb:3
                   node_call      times in test_segv.rb:3
                   node_vcall     test in test_segv.rb:9
(gdb) ruby threads list
0x15890 main      thread   THREAD_STOPPED    WAIT_JOIN(0x19ef400)    4417   bytes
0x19ef4           thread   THREAD_STOPPED    WAIT_TIME(57.10)        6267   bytes
0x19e34           thread   THREAD_STOPPED    WAIT_FD(5)             10405   bytes
0x19dc4           thread   THREAD_STOPPED    WAIT_NONE              14237   bytes
0x19dc8           thread   THREAD_STOPPED    WAIT_NONE              14237   bytes
0x19dcc           thread   THREAD_STOPPED    WAIT_NONE              14237   bytes
0x22668           thread   THREAD_STOPPED    WAIT_NONE              14237   bytes
0x1d630      curr thread   THREAD_RUNNABLE   WAIT_NONE

    (gdb) ruby eval 1+2                         (gdb) ruby objects
    3                                             HEAPS            8
    (gdb) ruby eval Thread.current                SLOTS      1686252
    #<Thread:0x1d630 run>                         LIVE        893327 (52.98%)
    (gdb) ruby eval Thread.list.size              FREE        792925 (47.02%)
    8
                                                  scope         1641   (0.18%)
    (gdb) ruby objects strings                    regexp        2255   (0.25%)
          140 u'lib'                              data          3539   (0.40%)
          158 u'0'                                class         3680   (0.41%)
          294 u'n'                               hash          6196   (0.69%)
          619 u''                                 object        8785   (0.98%)
                                                  array        13850   (1.55%)
        30503 unique strings                      string      105350   (11.79%)
      3187435 bytes                               node        742346   (83.10%)
rails_warden leak
(gdb) ruby objects classes
    1197 MIME::Type
    2657 NewRelic::MetricSpec
    2719 TZInfo::TimezoneTransitionInfo
    4124 Warden::Manager
    4124 MethodOverrideForAll
    4124 AccountMiddleware
    4124 Rack::Cookies
    4125 ActiveRecord::ConnectionAdapters::ConnectionManagement
    4125 ActionController::Session::CookieStore
    4125 ActionController::Failsafe
    4125 ActionController::ParamsParser
    4125 Rack::Lock
    4125 ActionController::Dispatcher
    4125 ActiveRecord::QueryCache
    4125 ActiveSupport::MessageVerifier
    4125 Rack::Head


middleware chain leaking per request
mongrel sleeper thread
0x16814c00         thread THREAD_STOPPED    WAIT_TIME(0.47) 1522 bytes
      node_fcall   sleep in lib/mongrel/configurator.rb:285
      node_fcall   run in lib/mongrel/configurator.rb:285
      node_fcall   loop in lib/mongrel/configurator.rb:285
      node_call    run in lib/mongrel/configurator.rb:285
      node_call    initialize in lib/mongrel/configurator.rb:285
      node_call    new in lib/mongrel/configurator.rb:285
      node_call    run in bin/mongrel_rails:128
      node_call    run in lib/mongrel/command.rb:212
      node_call    run in bin/mongrel_rails:281
      node_fcall   (unknown) in bin/mongrel_rails:19

    def run
      @listeners.each {|name,s|
        s.run
      }

      $mongrel_sleeper_thread = Thread.new { loop { sleep 1 } }
    end
god memory leaks
(gdb) ruby objects arrays
 elements instances
                             43   God::Process
    94310 3
                             43   God::Watch
    94311 3
                             43   God::Driver
    94314 2
                             43   God::DriverEventQueue
    94316 1
                             43   God::Conditions::MemoryUsage
                             43   God::Conditions::ProcessRunning
     5369 arrays
                             43   God::Behaviors::CleanPidFile
  2863364 member elements
                             45   Process::Status
                             86   God::Metric
many arrays with            327   God::System::SlashProcPoller
 90k+ elements!             327   God::System::Process
                            406   God::DriverEvent



   5 separate god leaks fixed by Eric
    Lindvall with the help of gdb.rb!
ruby method cache
                                                 (gdb) ruby methodcache
Module#extend wipes the                           UnboundMethod#arity
                                                  Hash#[]=
 entire method cache!                             Class#private
                                                  Array#freeze
(gdb) b rb_clear_cache                            Module#Integer
Breakpoint 1 at 0x41067b: file eval.c, line 351. Fixnum#<
(gdb) c                                           Class#is_a?
Continuing.                                       Fixnum#+
                                                  Class#protected
Breakpoint 1, rb_clear_cache () at eval.c:351     Class#>=
351	     if (!ruby_running) return;
(gdb) ruby threads                               2028 empty slots (99.02%)

0x1623000 main curr thread THREAD_RUNNABLE WAIT_NONE
   node_call      extend_object in sin.rb:23
   node_call      extend in sin.rb:23
   node_call      GET /other in lib/sinatra/base.rb:779
   node_call      GET /other in lib/sinatra/base.rb:779
   node_call      call in lib/sinatra/base.rb:779
   node_fcall     route in lib/sinatra/base.rb:474
bleak_house
       ruby memory leak detector

               gem install bleak_house


export RUBYOPT=”-r`gem which bleak_house | tail -1`”

              ruby-bleak-house myapp.rb

           bleak /tmp/bleak.5979.000.dump



      http://github.com/fauna/bleak_house
191691 total objects
Final heap size 191691 filled, 220961 free
Displaying top 20 most common line/class pairs
89513 __null__:__null__:__node__
41438 __null__:__null__:String
2348 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:Array
1508 lib/ruby/gems/1.8/specifications/gettext-1.9.gemspec:14:String
1021 lib/ruby/gems/1.8/specifications/heel-0.2.0.gemspec:14:String
 951 lib/ruby/site_ruby/1.8/rubygems/version.rb:111:String
 935 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:String
 834 lib/ruby/site_ruby/1.8/rubygems/version.rb:146:Array


      BleakHouse

         installs a patched version of ruby: ruby-bleak-
         house

         unlike gdb.rb, see where objects were created
         (file:line)

         create multiple dumps over time with `kill -USR2
         <pid>` and compare to find leaks
Coming soon:
bleak_house++
  no patches or need to recompile
  ruby
  “assembly metaprogramming” to
  setup a trampoline on rb_new_obj
  http://timetobleed.com/rewrite-
  your-ruby-vm-at-runtime-to-hot-
  patch-useful-features/

memory profiler
  coredump a production ruby process
  load the core to generate profiles
  of memory usage and leaks
Questions?

  @joedamato           @tmm1

timetobleed.com   github.com/tmm1



      Thanks for listening!

More Related Content

What's hot

Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
Jarod Wang
 
David container security-with_falco
David container security-with_falcoDavid container security-with_falco
David container security-with_falco
Lorenzo David
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPF
Brendan Gregg
 
Практический опыт профайлинга и оптимизации производительности Ruby-приложений
Практический опыт профайлинга и оптимизации производительности Ruby-приложенийПрактический опыт профайлинга и оптимизации производительности Ruby-приложений
Практический опыт профайлинга и оптимизации производительности Ruby-приложений
Olga Lavrentieva
 
ch6-pv2-device-drivers
ch6-pv2-device-driversch6-pv2-device-drivers
ch6-pv2-device-drivers
yushiang fu
 

What's hot (20)

Deploying Prometheus stacks with Juju
Deploying Prometheus stacks with JujuDeploying Prometheus stacks with Juju
Deploying Prometheus stacks with Juju
 
Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
 
Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021
Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021
Tracing MariaDB server with bpftrace - MariaDB Server Fest 2021
 
Osol Pgsql
Osol PgsqlOsol Pgsql
Osol Pgsql
 
Pledge in OpenBSD
Pledge in OpenBSDPledge in OpenBSD
Pledge in OpenBSD
 
A little systemtap
A little systemtapA little systemtap
A little systemtap
 
David container security-with_falco
David container security-with_falcoDavid container security-with_falco
David container security-with_falco
 
Performance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux KernelPerformance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux Kernel
 
Rac introduction
Rac introductionRac introduction
Rac introduction
 
計算機性能の限界点とその考え方
計算機性能の限界点とその考え方計算機性能の限界点とその考え方
計算機性能の限界点とその考え方
 
Linux Tracing Superpowers by Eugene Pirogov
Linux Tracing Superpowers by Eugene PirogovLinux Tracing Superpowers by Eugene Pirogov
Linux Tracing Superpowers by Eugene Pirogov
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPF
 
True stories on the analysis of network activity using Python
True stories on the analysis of network activity using PythonTrue stories on the analysis of network activity using Python
True stories on the analysis of network activity using Python
 
SSL Failing, Sharing, and Scheduling
SSL Failing, Sharing, and SchedulingSSL Failing, Sharing, and Scheduling
SSL Failing, Sharing, and Scheduling
 
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernelKernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
 
BPF / XDP 8월 세미나 KossLab
BPF / XDP 8월 세미나 KossLabBPF / XDP 8월 세미나 KossLab
BPF / XDP 8월 세미나 KossLab
 
Практический опыт профайлинга и оптимизации производительности Ruby-приложений
Практический опыт профайлинга и оптимизации производительности Ruby-приложенийПрактический опыт профайлинга и оптимизации производительности Ruby-приложений
Практический опыт профайлинга и оптимизации производительности Ruby-приложений
 
370410176 moshell-commands
370410176 moshell-commands370410176 moshell-commands
370410176 moshell-commands
 
Linux seccomp(2) vs OpenBSD pledge(2)
Linux seccomp(2) vs OpenBSD pledge(2)Linux seccomp(2) vs OpenBSD pledge(2)
Linux seccomp(2) vs OpenBSD pledge(2)
 
ch6-pv2-device-drivers
ch6-pv2-device-driversch6-pv2-device-drivers
ch6-pv2-device-drivers
 

Viewers also liked

Debugging Ruby (Aman Gupta)
Debugging Ruby (Aman Gupta)Debugging Ruby (Aman Gupta)
Debugging Ruby (Aman Gupta)
MongoSF
 
Rubinius @ RubyAndRails2010
Rubinius @ RubyAndRails2010Rubinius @ RubyAndRails2010
Rubinius @ RubyAndRails2010
Dirkjan Bussink
 
Introduction to tempest
Introduction to tempest Introduction to tempest
Introduction to tempest
openstackindia
 

Viewers also liked (20)

Debugging Ruby Systems
Debugging Ruby SystemsDebugging Ruby Systems
Debugging Ruby Systems
 
Debugging Ruby (Aman Gupta)
Debugging Ruby (Aman Gupta)Debugging Ruby (Aman Gupta)
Debugging Ruby (Aman Gupta)
 
Rubinius @ RubyAndRails2010
Rubinius @ RubyAndRails2010Rubinius @ RubyAndRails2010
Rubinius @ RubyAndRails2010
 
OpenSource Hardware -Debian Way
OpenSource Hardware -Debian WayOpenSource Hardware -Debian Way
OpenSource Hardware -Debian Way
 
Operating OPNFV
Operating OPNFVOperating OPNFV
Operating OPNFV
 
OpenStack@Mini-Deb Conf'16 Mumbai
OpenStack@Mini-Deb Conf'16 MumbaiOpenStack@Mini-Deb Conf'16 Mumbai
OpenStack@Mini-Deb Conf'16 Mumbai
 
Copr HD OpenStack Day India
Copr HD OpenStack Day IndiaCopr HD OpenStack Day India
Copr HD OpenStack Day India
 
Your first patch to OpenStack
Your first patch to OpenStackYour first patch to OpenStack
Your first patch to OpenStack
 
Deploying openstack using ansible
Deploying openstack using ansibleDeploying openstack using ansible
Deploying openstack using ansible
 
The OpenStack Contribution Workflow
The OpenStack Contribution WorkflowThe OpenStack Contribution Workflow
The OpenStack Contribution Workflow
 
Your first patch to open stack
Your first patch to open stackYour first patch to open stack
Your first patch to open stack
 
Open stack qa and tempest
Open stack qa and tempestOpen stack qa and tempest
Open stack qa and tempest
 
Guts & OpenStack migration
Guts & OpenStack migrationGuts & OpenStack migration
Guts & OpenStack migration
 
OpenStack Storage Buddy Ceph
OpenStack Storage Buddy CephOpenStack Storage Buddy Ceph
OpenStack Storage Buddy Ceph
 
Introduction to tempest
Introduction to tempest Introduction to tempest
Introduction to tempest
 
20150509 unix v6로 배우는 커널의 원리와 구조 3 김지은
20150509 unix v6로 배우는 커널의 원리와 구조 3 김지은20150509 unix v6로 배우는 커널의 원리와 구조 3 김지은
20150509 unix v6로 배우는 커널의 원리와 구조 3 김지은
 
Pgcon2012 ori-20120224
Pgcon2012 ori-20120224Pgcon2012 ori-20120224
Pgcon2012 ori-20120224
 
Ryu with OpenFlow 1.3, Traffic Monitor
Ryu with OpenFlow 1.3, Traffic MonitorRyu with OpenFlow 1.3, Traffic Monitor
Ryu with OpenFlow 1.3, Traffic Monitor
 
Who carries your container? Zun or Magnum?
Who carries your container? Zun or Magnum?Who carries your container? Zun or Magnum?
Who carries your container? Zun or Magnum?
 
OPNFV & OpenStack
OPNFV & OpenStackOPNFV & OpenStack
OPNFV & OpenStack
 

Similar to Debugging Ruby

Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and Architecture
Sidney Chen
 
Crash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_TizenCrash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_Tizen
Lex Yu
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
Brendan Gregg
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
Brendan Gregg
 
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
Anne Nicolas
 

Similar to Debugging Ruby (20)

Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
 
Reverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande ModemReverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande Modem
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringOSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
 
A little systemtap
A little systemtapA little systemtap
A little systemtap
 
A close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issuesA close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issues
 
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and Architecture
 
Crash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_TizenCrash_Report_Mechanism_In_Tizen
Crash_Report_Mechanism_In_Tizen
 
pstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle databasepstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle database
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
 
Kernel crashdump
Kernel crashdumpKernel crashdump
Kernel crashdump
 
Debugging linux issues with eBPF
Debugging linux issues with eBPFDebugging linux issues with eBPF
Debugging linux issues with eBPF
 
hacking-embedded-devices.pptx
hacking-embedded-devices.pptxhacking-embedded-devices.pptx
hacking-embedded-devices.pptx
 
Hacking the swisscom modem
Hacking the swisscom modemHacking the swisscom modem
Hacking the swisscom modem
 
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
Kernel Recipes 2017 - Performance analysis Superpowers with Linux BPF - Brend...
 
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
 
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation CenterDUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
 
Open Source Systems Performance
Open Source Systems PerformanceOpen Source Systems Performance
Open Source Systems Performance
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Peter Udo Diehl
 

Recently uploaded (20)

Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 

Debugging Ruby

  • 1. Debugging Ruby troubleshooting the VM and your app Aman Gupta
  • 2. About Aman Gupta San Francisco, CA Ruby Performance Consulting (Twitter, Github, Heroku, ZenDesk) Ruby Hero 2009 EventMachine, amqp, REE, sinbook, perftools.rb, gdb.rb github.com/tmm1 @tmm1
  • 3. The Ruby VM MRI- Ruby 1.8.{6,7} REE- Ruby 1.8.7 + patches Tools for C code Tested primarily on: Linux operating system Intel CPUs (i686 and x86_64)
  • 5. lsof -nPp <pid> -n Inhibits the conversion of network numbers to host names for network files. -P Inhibits the conversion of port numbers to port names for network files FD TYPE SIZE NODE NAME json cwd DIR 4096 10437557 /var/www/myapp memcached rtd DIR 4096 2 / mysql txt REG 1925210 1089684 /usr/bin/ruby mem REG 48188 7061523 /json-1.1.9/ext/json/ext/generator.so http mem REG 46970 7061524 /json-1.1.9/ext/json/ext/parser.so mem REG 3428025 1131339 /memcached-0.17.4/lib/rlibmemcached.so mem REG 152972 5303443 /mysql-2.8.1/lib/mysql_api.so mem REG 154013 1089708 /usr/lib/libtcmalloc_minimal.so.0.0.0 mem REG 119288 11616294 /lib/ld-2.7.so 0u CHR 303 /dev/null 1w REG 4746405529 1106245 /usr/local/nginx/logs/error.log 2w REG 4746405529 1106245 /usr/local/nginx/logs/error.log 3u IPv4 TCP 10.8.85.66:33326->10.8.85.68:3306 (ESTABLISHED) 10u IPv4 TCP 10.8.85.66:33327->10.8.85.68:3306 (ESTABLISHED) 11u IPv4 TCP 127.0.0.1:58273->127.0.0.1:11211 (ESTABLISHED) 12u REG 20182 12463107 /tmp/RackMultipart.28957.0 33u IPv4 TCP 174.36.83.42:37466->69.63.180.21:80 (ESTABLISHED)
  • 6. strace trace system calls and signals strace -cp <pid> strace -ttTp <pid> -o <file>
  • 7. strace -cp <pid> -c Count time, calls, and errors for each system call and report a summary on program exit. -p pid Attach to the process with the process ID pid and begin tracing. % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 50.39 0.000064 0 1197 592 read 34.65 0.000044 0 609 writev 14.96 0.000019 0 1226 epoll_ctl 0.00 0.000000 0 4 close 0.00 0.000000 0 1 select 0.00 0.000000 0 4 socket 0.00 0.000000 0 4 4 connect 0.00 0.000000 0 1057 epoll_wait ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000127 4134 596 total
  • 8. strace -ttTp <pid> -o <file> -t Prefix each line of the trace with the time of day. -tt If given twice, the time printed will include the microseconds. -T Show the time spent in system calls. This records the time difference between the beginning and the end of each system call. -o filename Write the trace output to the file filename rather than to stderr. 01:09:11.266949 epoll_wait(9, {{EPOLLIN, {u32=68841296, u64=68841296}}}, 4096, 50) = 1 <0.033109> 01:09:11.300102 accept(10, {sa_family=AF_INET, sin_port=38313, sin_addr="127.0.0.1"}, [1226]) = 22 <0.000014> 01:09:11.300190 fcntl(22, F_GETFL) = 0x2 (flags O_RDWR) <0.000007> 01:09:11.300237 fcntl(22, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000008> 01:09:11.300277 setsockopt(22, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008> 01:09:11.300489 accept(10, 0x7fff5d9c07d0, [1226]) = -1 EAGAIN <0.000014> 01:09:11.300547 epoll_ctl(9, EPOLL_CTL_ADD, 22, {EPOLLIN, {u32=108750368, u64=108750368}}) = 0 <0.000009> 01:09:11.300593 epoll_wait(9, {{EPOLLIN, {u32=108750368, u64=108750368}}}, 4096, 50) = 1 <0.000007> 01:09:11.300633 read(22, "GET / HTTP/1.1r"..., 16384) = 772 <0.000012> 01:09:11.301727 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000007> 01:09:11.302095 poll([{fd=5, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout) <0.000008> 01:09:11.302144 write(5, "1000000-0003SELECT * FROM `table`"..., 56) = 56 <0.000023> 01:09:11.302221 read(5, "25101,20x234m"..., 16384) = 284 <1.300897>
  • 9. stracing ruby: SIGVTALRM 15:45:51.658164 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.658244 rt_sigreturn(0x1a) = 2207807 <0.000009> 15:45:51.678208 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.678271 rt_sigreturn(0x1a) = 0 <0.000009> 15:45:51.698161 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.698216 rt_sigreturn(0x1a) = 140734552062624 <0.000009> 15:45:51.718154 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.718192 rt_sigreturn(0x1a) = 140734552066688 <0.000009> 15:45:51.738185 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.738221 rt_sigreturn(0x1a) = 11333952 <0.000008> 15:45:51.758162 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.758216 rt_sigreturn(0x1a) = 0 <0.000009> 15:45:51.818168 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- 15:45:51.819817 rt_sigreturn(0x1a) = 1 <0.000010> 15:45:51.838196 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- ruby 1.8 uses signals to schedule its green threads process receives a SIGVTALRM signal every 10ms
  • 10. stracing ruby: sigprocmask % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 0.326334 0 3568567 rt_sigprocmask 0.00 0.000000 0 9 read 0.00 0.000000 0 10 open 0.00 0.000000 0 10 close 0.00 0.000000 0 9 fstat 0.00 0.000000 0 25 mmap ------ ----------- ----------- --------- --------- ---------------- 100.00 0.326334 3568685 0 total debian/redhat compile ruby with --enable-pthread uses a native thread timer for SIGVTALRM causes excessive calls to sigprocmask: 30% slowdown!
  • 11. tcpdump dump traffic on a network tcpdump -i eth1 -s 1500 -nqA tcp dst port 80 tcpdump -i eth0 -s 1500 -nqA tcp dst port 3306 tcpdump -i eth1 -s 1500 -w <file> tcp dst port 80
  • 12. tcpdump -i <eth> -s <len> -nqA <expr> tcpdump -i <eth> -w <file> <expr> -i <eth> Network interface. -s <len> Snarf len bytes of data from each packet. -n Don't convert addresses (host addresses, port numbers) to names. -q Quiet output. Print less protocol information. -A Print each packet (minus its link level header) in ASCII. -w <file> Write the raw packets to file rather than printing them out. <expr> libpcap expression, for example: tcp src port 80 tcp dst port 3306
  • 13. tcp dst port 80 19:52:20.216294 IP 24.203.197.27.40105 > 174.37.48.236.80: tcp 438 E...*.@.l.%&.....%0....POx..%s.oP.......GET /poll_images/cld99erh0/logo.png HTTP/1.1 Accept: */* Referer: http://apps.facebook.com/realpolls/?_fb_q=1 tcp dst port 3306 19:51:06.501632 IP 10.8.85.66.50443 > 10.8.85.68.3306: tcp 98 E..."K@.@.Yy .UB .UD.....z....L............ GZ.y3b..[......W....SELECT * FROM `votes` WHERE (`poll_id` = 72621) LIMIT 1 tcpdump -w <file>
  • 14. google-perftools Google’s CPU profiler export LD_PRELOAD=libprofiler.so export DYLD_INSERT_LIBRARIES=libprofiler.dylib CPUPROFILE=/tmp/myprof ./myapp pprof ./myapp /tmp/myprof
  • 15. wget http://google-perftools.googlecode.com/files/google- perftools-1.4.tar.gz download tar zxvf google-perftools-1.4.tar.gz cd google-perftools-1.4 ./configure --prefix=/opt make compile sudo make install # for linux export LD_PRELOAD=/opt/lib/libprofiler.so setup # for osx export DYLD_INSERT_LIBRARIES=/opt/lib/libprofiler.dylib CPUPROFILE=/tmp/ruby.prof ruby -e' profile 5_000_000.times{ "hello world" } ' pprof `which ruby` --text /tmp/ruby.prof report
  • 16. pprof ruby pprof ruby ruby.prof --text ruby.prof --gif Total: 103 samples 20 19.4% 19.4% 95 92.2% rb_yield_0 11 10.7% 30.1% 103 100.0% rb_eval 8 7.8% 37.9% 12 11.7% gc_sweep 3 2.9% 68.9% 52 50.5% rb_str_new3 3 2.9% 74.8% 3 2.9% obj_free 3 2.9% 77.7% 103 100.0% int_dotimes 3 2.9% 80.6% 12 11.7% gc_mark
  • 17. Profiling MRI 10% of production VM time spent in rb_str_sub_bang String#sub! called from Time.parse return unless str.sub!(/A(d{1,2})/, '') return unless str.sub!(/A( d|d{1,2})/, '') return unless str.sub!(/A( d|d{1,2})/, '') return unless str.sub!(/A(d{1,3})/, '') return unless str.sub!(/A(d{1,2})/, '') return unless str.sub!(/A(d{1,2})/, '') switch to third_base gem ThirdBase: Fast and Easy Date/DateTime class for Ruby
  • 18. Profiling EM + threads Total: 3763 samples 2764 73.5% catch_timer 989 26.3% memcpy 3 0.1% st_lookup 2 0.1% rb_thread_schedule 1 0.0% rb_eval 1 0.0% rb_newobj 1 0.0% rb_gc_force_recycle known issue: EM+threads = slow memcpy?? thread context switches copy the stack w/ memcpy EM allocates huge buffer on the stack solution: move buffer to the heap
  • 19. ltrace trace library calls ltrace -cp <pid> ltrace -ttTp <pid> -o <file>
  • 20. ltrace -c ruby threaded_em.rb % time seconds usecs/call calls function ------ ----------- ----------- --------- -------------------- 48.65 11.741295 617 19009 memcpy 30.16 7.279634 831 8751 longjmp 9.78 2.359889 135 17357 _setjmp 8.91 2.150565 285 7540 malloc 1.10 0.265946 20 13021 memset 0.81 0.195272 19 10105 __ctype_b_loc 0.35 0.084575 19 4361 strcmp 0.19 0.046163 19 2377 strlen 0.03 0.006272 23 265 realloc ------ ----------- ----------- --------- -------------------- 100.00 24.134999 82999 total ltrace -ttT -e memcpy ruby threaded_em.rb 01:24:48.769408 --- SIGVTALRM (Virtual timer expired) --- 01:24:48.769616 memcpy(0x1216000, "", 1086328) = 0x1216000 <0.000578> 01:24:48.770555 memcpy(0x6e32670, "240&343v", 1086328) = 0x6e32670 <0.000418> 01:24:49.899414 --- SIGVTALRM (Virtual timer expired) --- 01:24:49.899490 memcpy(0x1320000, "", 1082584) = 0x1320000 <0.000628> 01:24:49.900474 memcpy(0x6e32670, "", 1086328) = 0x6e32670 <0.000479>
  • 21. ltrace/libdl trace dlopen’d library calls ltrace -F <conf> -bg -x <symbol> -p <pid> http://github.com/ice799/ltrace/tree/libdl
  • 22. ltrace -F <conf> -b -g -x <sym> -b Ignore signals. -g Ignore libraries linked at compile time. -F <conf> Read prototypes from config file. -x <sym> Trace calls to the function sym. -s <num> Show first num bytes of string args. -F ltrace.conf int mysql_real_query(addr,string,ulong); void garbage_collect(void); int memcached_set(addr,string,ulong,string,ulong);
  • 23. ltrace -x garbage_collect 19:08:06.436926 garbage_collect() = <void> <0.221679> 19:08:15.329311 garbage_collect() = <void> <0.187546> 19:08:17.662149 garbage_collect() = <void> <0.199200> 19:08:20.486655 garbage_collect() = <void> <0.205864> 19:08:25.102302 garbage_collect() = <void> <0.214295> 19:08:35.552337 garbage_collect() = <void> <0.189172> ltrace -x mysql_real_query 19:09:11.493395 mysql_real_query(0x19c7a500, "SELECT * FROM `users`", 21) = 0 <1.206506> 19:09:16.630981 mysql_real_query(0x1c9e0500, "SET NAMES 'UTF8'", 16) = 0 <0.000324> 19:09:16.631446 mysql_real_query(0x1c9e0500, "SET SQL_AUTO_IS_NULL=0", 22) = 0 <0.000322> 19:09:16.654231 mysql_real_query(0x1c9e0500, "COMMIT", 6) = 0 <0.000181> ltrace -x memcached_set memcached_set(0x15d46b80, "Status:5456561633", 21, "004bo:01{", 366) = 0 <0.001116> memcached_set(0x15d46b80, "Status:5453277696", 21, "004bo:01{", 333) = 0 <0.000224> memcached_set(0x15d46b80, "Status:5435377757", 21, "004bo:01{", 298) = 0 <0.001850> memcached_set(0x15d46b80, "Status:5435122010", 21, "004bo:01{", 302) = 0 <0.000530> memcached_set(0x15d46b80, "Status:5407037167", 21, "004bo:01{", 318) = 0 <0.000291> memcached_set(0x15d46b80, "Status:5405690802", 21, "004bo:01{", 299) = 0 <0.000658> memcached_set(0x15d46b80, "Status:5343957534", 21, "004bo:01{", 264) = 0 <0.000243>
  • 24. gdb the GNU debugger gdb <program> gdb <program> <pid> Be sure to build with: -ggdb -O0
  • 25. gdb walkthrough % gdb ./test-it start gdb (gdb) b average set breakpoint on function named average Breakpoint 1 at 0x1f8e: file test-it.c, line 3. (gdb) run run program Starting program: /Users/joe/test-it Reading symbols for shared libraries ++. done Breakpoint 1, average (x=5, y=6) at test-it.c:3 hit breakpoint! 3 int sum = x + y; (gdb) bt show backtrace #0 average (x=5, y=6) at test-it.c:3 function stack #1 0x00001fec in main () at test-it.c:12 (gdb) s 4 double avg = sum / 2.0; single step (gdb) s 5 return avg; (gdb) p avg $1 = 5.5 print variables (gdb) p sum $2 = 11
  • 26. Ruby VM stack traces (gdb) where #0 0x0002a55e in rb_call (klass=1386800, recv=5056455, mid=42, argc=1, argv=0xbfffe5c0, scope=0, self=1403220) at eval.c:6125 #1 0x000226ef in rb_eval (self=1403220, n=0x1461e4) at eval.c:3493 #2 0x00026d01 in rb_yield_0 (val=5056455, self=1403220, klass=0, flags=0, avalue=0) at eval.c:5083 #3 0x000270e8 in rb_yield (val=5056455) at eval.c:5168 #4 0x0005c30c in int_dotimes (num=1000000001) at numeric.c:2946 #5 0x00029be3 in call_cfunc (func=0x5c2a0 <int_dotimes>, recv=1000000001, len=0, argc=0, argv=0x0) at eval.c:5759 #6 0x00028fd4 in rb_call0 (klass=1387580, recv=1000000001, id=5785, oid=5785, argc=0, argv=0x0, body=0x152b24, flags=0) at eval.c:5911 #7 0x0002a7a7 in rb_call (klass=1387580, recv=1000000001, mid=5785, argc=0, argv=0x0, scope=0, self=1403220) at eval.c:6158 #8 0x000226ef in rb_eval (self=1403220, n=0x146284) at eval.c:3493 #9 0x000213e3 in rb_eval (self=1403220, n=0x1461a8) at eval.c:3223 #10 0x0001ceea in eval_node (self=1403220, node=0x1461a8) at eval.c:1437 #11 0x0001d60f in ruby_exec_internal () at eval.c:1642 #12 0x0001d660 in ruby_exec () at eval.c:1662 #13 0x0001d68e in ruby_run () at eval.c:1672 #14 0x000023dc in main (argc=2, argv=0xbffff7c4, envp=0xbffff7d0) at main.c:48 rb_eval recursively executes ruby code in 1.8
  • 27. Debugging Ruby Segfaults test_segv.rb:4: [BUG] Segmentation fault ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0] def test #include "ruby.h" require 'segv' 4.times do VALUE Dir.chdir '/tmp' do segv() Hash.new{ segv }[0] { end VALUE array[1]; end array[1000000] = NULL; end return Qnil; } sleep 10 test() void Init_segv() { rb_define_method(rb_cObject, "segv", segv, 0); }
  • 28. 1. Attach to running process $ ps aux | grep ruby joe 23611 0.0 0.1 25424 7540 S Dec01 0:00 ruby test_segv.rb $ sudo gdb ruby 23611 Attaching to program: ruby, process 23611 0x00007fa5113c0c93 in nanosleep () from /lib/libc.so.6 (gdb) c Continuing. Program received signal SIGBUS, Bus error. segv () at segv.c:7 7 array[1000000] = NULL; 2. Use a coredump Process.setrlimit Process::RLIMIT_CORE, 300*1024*1024 $ sudo mkdir /cores $ sudo chmod 777 /cores $ sudo sysctl kernel.core_pattern=/cores/%e.core.%s.%p.%t $ sudo gdb ruby /cores/ruby.core.6.23611.1259781224
  • 29. def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end end (gdb) where end #0 segv () at segv.c:7 #1 0x000000000041f2be in call_cfunc () at eval.c:5727 test() ... #13 0x000000000043ba8c in rb_hash_default () at hash.c:521 ... #19 0x000000000043b92a in rb_hash_aref () at hash.c:429 ... #26 0x00000000004bb7bc in chdir_yield () at dir.c:728 #27 0x000000000041d8d7 in rb_ensure () at eval.c:5528 #28 0x00000000004bb93a in dir_s_chdir () at dir.c:816 ... #35 0x000000000041c444 in rb_yield () at eval.c:5142 #36 0x0000000000450690 in int_dotimes () at numeric.c:2834 ... #48 0x0000000000412a90 in ruby_run () at eval.c:1678 #49 0x000000000041014e in main () at main.c:48
  • 31. perftools.rb google-perftools for ruby gem install perftools.rb export RUBYOPT=”-r`gem which perftools | tail -1`” CPUPROFILE=/tmp/myprof ruby myapp.rb pprof.rb /tmp/myprof http://github.com/tmm1/perftools.rb
  • 32. require 'sinatra' $ ab -c 1 -n 50 http://127.0.0.1:4567/compute $ ab -c 1 -n 50 http://127.0.0.1:4567/sleep get '/sleep' do sleep 0.25 Sampling profiler: 'done' end 232 samples total get '/compute' do 83 samples were in /compute proc{ |n| a,b=0,1 118 samples had /compute on n.times{ a,b = b,a+b } the stack but were in b another function }.call(10_000) 'done' /compute accounts for 50% of process, but only 35% of end time was in /compute itself == Sinatra has ended his set (crowd applauds) PROFILE: interrupts/evictions/bytes = 232/0/2152 Total: 232 samples 83 35.8% 35.8% 118 50.9% Sinatra::Application#GET /compute 56 24.1% 59.9% 56 24.1% garbage_collector 35 15.1% 75.0% 113 48.7% Integer#times
  • 33. CPUPROFILE_REALTIME=1 CPUPROFILE=app.prof CPUPROFILE=app-rt.prof
  • 36. gdb.rb gdb with MRI hooks gdb.rb <pid> http://github.com/tmm1/gdb.rb
  • 37. def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end end end (gdb) ruby threads test() 0xa3e000 main curr thread THREAD_RUNNABLE WAIT_NONE node_vcall segv in test_segv.rb:5 node_call test in test_segv.rb:5 node_call call in test_segv.rb:5 node_call default in test_segv.rb:5 node_call [] in test_segv.rb:5 node_call test in test_segv.rb:4 node_call chdir in test_segv.rb:4 node_call test in test_segv.rb:3 node_call times in test_segv.rb:3 node_vcall test in test_segv.rb:9
  • 38. (gdb) ruby threads list 0x15890 main thread THREAD_STOPPED WAIT_JOIN(0x19ef400) 4417 bytes 0x19ef4 thread THREAD_STOPPED WAIT_TIME(57.10) 6267 bytes 0x19e34 thread THREAD_STOPPED WAIT_FD(5) 10405 bytes 0x19dc4 thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x19dc8 thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x19dcc thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x22668 thread THREAD_STOPPED WAIT_NONE 14237 bytes 0x1d630 curr thread THREAD_RUNNABLE WAIT_NONE (gdb) ruby eval 1+2 (gdb) ruby objects 3 HEAPS 8 (gdb) ruby eval Thread.current SLOTS 1686252 #<Thread:0x1d630 run> LIVE 893327 (52.98%) (gdb) ruby eval Thread.list.size FREE 792925 (47.02%) 8 scope 1641 (0.18%) (gdb) ruby objects strings regexp 2255 (0.25%) 140 u'lib' data 3539 (0.40%) 158 u'0' class 3680 (0.41%) 294 u'n' hash 6196 (0.69%) 619 u'' object 8785 (0.98%) array 13850 (1.55%) 30503 unique strings string 105350 (11.79%) 3187435 bytes node 742346 (83.10%)
  • 39. rails_warden leak (gdb) ruby objects classes 1197 MIME::Type 2657 NewRelic::MetricSpec 2719 TZInfo::TimezoneTransitionInfo 4124 Warden::Manager 4124 MethodOverrideForAll 4124 AccountMiddleware 4124 Rack::Cookies 4125 ActiveRecord::ConnectionAdapters::ConnectionManagement 4125 ActionController::Session::CookieStore 4125 ActionController::Failsafe 4125 ActionController::ParamsParser 4125 Rack::Lock 4125 ActionController::Dispatcher 4125 ActiveRecord::QueryCache 4125 ActiveSupport::MessageVerifier 4125 Rack::Head middleware chain leaking per request
  • 40. mongrel sleeper thread 0x16814c00 thread THREAD_STOPPED WAIT_TIME(0.47) 1522 bytes node_fcall sleep in lib/mongrel/configurator.rb:285 node_fcall run in lib/mongrel/configurator.rb:285 node_fcall loop in lib/mongrel/configurator.rb:285 node_call run in lib/mongrel/configurator.rb:285 node_call initialize in lib/mongrel/configurator.rb:285 node_call new in lib/mongrel/configurator.rb:285 node_call run in bin/mongrel_rails:128 node_call run in lib/mongrel/command.rb:212 node_call run in bin/mongrel_rails:281 node_fcall (unknown) in bin/mongrel_rails:19 def run @listeners.each {|name,s| s.run } $mongrel_sleeper_thread = Thread.new { loop { sleep 1 } } end
  • 41. god memory leaks (gdb) ruby objects arrays elements instances 43 God::Process 94310 3 43 God::Watch 94311 3 43 God::Driver 94314 2 43 God::DriverEventQueue 94316 1 43 God::Conditions::MemoryUsage 43 God::Conditions::ProcessRunning 5369 arrays 43 God::Behaviors::CleanPidFile 2863364 member elements 45 Process::Status 86 God::Metric many arrays with 327 God::System::SlashProcPoller 90k+ elements! 327 God::System::Process 406 God::DriverEvent 5 separate god leaks fixed by Eric Lindvall with the help of gdb.rb!
  • 42. ruby method cache (gdb) ruby methodcache Module#extend wipes the UnboundMethod#arity Hash#[]= entire method cache! Class#private Array#freeze (gdb) b rb_clear_cache Module#Integer Breakpoint 1 at 0x41067b: file eval.c, line 351. Fixnum#< (gdb) c Class#is_a? Continuing. Fixnum#+ Class#protected Breakpoint 1, rb_clear_cache () at eval.c:351 Class#>= 351 if (!ruby_running) return; (gdb) ruby threads 2028 empty slots (99.02%) 0x1623000 main curr thread THREAD_RUNNABLE WAIT_NONE node_call extend_object in sin.rb:23 node_call extend in sin.rb:23 node_call GET /other in lib/sinatra/base.rb:779 node_call GET /other in lib/sinatra/base.rb:779 node_call call in lib/sinatra/base.rb:779 node_fcall route in lib/sinatra/base.rb:474
  • 43. bleak_house ruby memory leak detector gem install bleak_house export RUBYOPT=”-r`gem which bleak_house | tail -1`” ruby-bleak-house myapp.rb bleak /tmp/bleak.5979.000.dump http://github.com/fauna/bleak_house
  • 44. 191691 total objects Final heap size 191691 filled, 220961 free Displaying top 20 most common line/class pairs 89513 __null__:__null__:__node__ 41438 __null__:__null__:String 2348 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:Array 1508 lib/ruby/gems/1.8/specifications/gettext-1.9.gemspec:14:String 1021 lib/ruby/gems/1.8/specifications/heel-0.2.0.gemspec:14:String 951 lib/ruby/site_ruby/1.8/rubygems/version.rb:111:String 935 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:String 834 lib/ruby/site_ruby/1.8/rubygems/version.rb:146:Array BleakHouse installs a patched version of ruby: ruby-bleak- house unlike gdb.rb, see where objects were created (file:line) create multiple dumps over time with `kill -USR2 <pid>` and compare to find leaks
  • 45. Coming soon: bleak_house++ no patches or need to recompile ruby “assembly metaprogramming” to setup a trampoline on rb_new_obj http://timetobleed.com/rewrite- your-ruby-vm-at-runtime-to-hot- patch-useful-features/ memory profiler coredump a production ruby process load the core to generate profiles of memory usage and leaks
  • 46. Questions? @joedamato @tmm1 timetobleed.com github.com/tmm1 Thanks for listening!