There and Back Again

1. There and Back Again – or –How I Set out to Benchmark an Algorithm and Ended Up Fixing Ruby

2. About Me Name: Joshua Ballanco

3. About MeLocation: Ankara, Turkey

4. About MeEmployer:

5. About MeEmployer: Burnside Digital

6. WARNINGThis is an ADVANCED presentation. While you donot need to be an expert on the inner workings ofC Ruby, this talk does presume you are familiarwith concepts such as pointers, memoryallocation, and pages (no, not book pages,memory pages).If you are prone to motion sickness or becomenauseous at the sight of hexadecimal memoryaddresses, this talk may not be for you.

7. Prelude: How can Icontribute to Ruby?

8. Prelude: How can Icontribute to Ruby? Use It!

9. 4 Rs of Bug Reporting• Reproduce• Report• Reduce• Regress

10. Reproduce It!• Can you make the bug occur every time? most of the time? some of the time?• .irbrc, .gemrc, Other rc ﬁles• Environment and Environment Variables• rvm/rbenv• Ask a friend

11. Report It!• Ruby Bug Tracker: http://bugs.ruby-lang.org/• All steps to reproduce, bonus points for attachments• Ruby Version• Crash log (hint: look in Console.app)

12. Reduce It!• Eliminate libraries• Simple scripts are best• Shrink code (5 or fewer lines is ideal)

13. Regress It!• Try different Ruby versions• Better: try different Ruby releases• Best: git bisect!• Bonus: try different build settings

14. 5 Rs of Bug Reporting• Reproduce• Report• Reduce• Regress• Repair!

15. A long time ago in a faraway land...

20. A long time ago in a faraway land...for full backstory, see: http://blogs.burnsidedigital.com/

21. Disaster Strikes!> ruby delegation_bench.rbCall one method user system total realPre-delegate 4.940000 0.020000 4.960000 ( 4.954945)Post-delegate 0.060000 0.000000 0.060000 ( 0.060528)On Demand 0.010000 0.000000 0.010000 ( 0.010604)Reversible 0.110000 0.000000 0.110000 ( 0.101494)Call one method 100 times...Call three methods 100 times user system total realPre-delegate 5.640000 0.000000 5.640000 ( 5.640255)Post-delegate 0.980000 0.000000 0.980000 ( 0.975919)On Demand 1.210000 0.000000 1.210000 ( 1.205214)Reversible 1.020000 0.000000 1.020000 ( 1.027759)Call three methods 10,000 times user system total realPre-delegate ruby(5919,0x7fff732a5180) malloc: *** error for object 0x7fb18f80d750:incorrect checksum for freed object - object was probably modified after being freed.*** set a breakpoint in malloc_error_break to debugzsh: abort ruby delegation_bench.rb

22. Disaster Strikes!

23. Pack Your Bags• Download Ruby Source (https://github.com/ruby/ruby)• Build ./configure && make (see ./configure --help for more)

24. Study the Map• ./include/ruby/ruby.h• ID => symbols• VALUE => everything else• rb_id2name => dump symbols in GDB• rb_string_value_cstr => #to_s for GDB• vm_call_method

25. Bring Your Gardener miniruby• like ruby, only smaller• lexer, parser, interpreter,VM/runtime• core library classes• make miniruby to build

26. Weapon of Choice gdb• gdb ./miniruby• Set pre-run breakpoints• Pass arguments to miniruby with “run”• Watch it crash!

27. > gdb ./miniruby...(gdb) run ./bug.rbStarting program: /Users/jballanc/Source/ruby/miniruby ./bug.rbReading symbols for shared libraries ++............................ doneProgram received signal EXC_BAD_ACCESS, Could not access memory.Reason: KERN_INVALID_ADDRESS at address: 0x00000011003971980x00007fff8e611cfb in tiny_free_list_remove_ptr ()(gdb)

30. (gdb) bt#0 0x00007fff8e611cfb in tiny_free_list_remove_ptr ()#1 0x00007fff8e60e76d in szone_free_definite_size ()#2 0x00007fff8e608898 in free ()#3 0x000000010012f5dc in rb_free_method_entry [inlined] () at vm_method.c:155#4 0x000000010012f5dc in rb_sweep_method_entry (pvm=0xb592caed) at...#5 0x000000010004a00a in gc_lazy_sweep [inlined] () at gc.c:2107#6 0x000000010004a00a in rb_newobj () at gc.c:1183#7 0x000000010000caf4 in ary_alloc [inlined] () at array.c:301#8 0x000000010000caf4 in ary_new (klass=4303833400, capa=3046355020) at...#9 0x000000010000d165 in rb_ary_new2 [inlined] () at array.c:334#10 0x000000010000d165 in rb_ary_new4 (n=0, elts=0x7fff5fbfee90) at array.c...#11 0x000000010013288a in vm_yield_with_cfunc (th=0x100303e90, block=...#12 0x000000010013dbb1 in rb_vm_invoke_proc (th=0x100303e90, proc=...#13 0x000000010013bd90 in vm_call_bmethod [inlined] () at vm_insnhelper.c:433#14 0x000000010013bd90 in vm_call_method (th=0x7fff5fbfef60,cfp=0x7fff5fbfef60, num=8766224, blockptr=0x7fff5fbfef60,flag=140734799802208, id=4298129040, me=0x10038ec90, recv=4303733520) atvm_insnhelper.c:566...(gdb) p rb_string_value_cstr(4303733520)

33. (gdb) p rb_string_value_cstr(4303733520)./bug.rb:7: [BUG] object allocation during garbage collection phaseruby 1.9.3p286 (2012-10-12 revision 36243) [x86_64-darwin12.2.0]-- Control frame information -----------------------------------------------c:0009 p:0011 s:0021 b:0021 l:000888 d:000020 BLOCK ./bug.rb:7c:0008 p:---- s:0019 b:0019 l:000018 d:000018 FINISHc:0007 p:---- s:0017 b:0017 l:000016 d:000016 CFUNC :times...

34. (gdb) p rb_string_value_cstr(4303733520)./bug.rb:7: [BUG] object allocation during garbage collection phaseruby 1.9.3p286 (2012-10-12 revision 36243) [x86_64-darwin12.2.0]-- Control frame information -----------------------------------------------c:0009 p:0011 s:0021 b:0021 l:000888 d:000020 BLOCK ./bug.rb:7c:0008 p:---- s:0019 b:0019 l:000018 d:000018 FINISHc:0007 p:---- s:0017 b:0017 l:000016 d:000016 CFUNC :times...

35. (gdb) bt#0 0x00007fff8e611cfb in tiny_free_list_remove_ptr ()#1 0x00007fff8e60e76d in szone_free_definite_size ()#2 0x00007fff8e608898 in free ()#3 0x000000010012f5dc in rb_free_method_entry [inlined] () at vm_method.c:155#4 0x000000010012f5dc in rb_sweep_method_entry (pvm=0xb592caed) at...#5 0x000000010004a00a in gc_lazy_sweep [inlined] () at gc.c:2107#6 0x000000010004a00a in rb_newobj () at gc.c:1183#7 0x000000010000caf4 in ary_alloc [inlined] () at array.c:301#8 0x000000010000caf4 in ary_new (klass=4303833400, capa=3046355020) at...#9 0x000000010000d165 in rb_ary_new2 [inlined] () at array.c:334#10 0x000000010000d165 in rb_ary_new4 (n=0, elts=0x7fff5fbfee90) at array.c...#11 0x000000010013288a in vm_yield_with_cfunc (th=0x100303e90, block=...#12 0x000000010013dbb1 in rb_vm_invoke_proc (th=0x100303e90, proc=...#13 0x000000010013bd90 in vm_call_bmethod [inlined] () at vm_insnhelper.c:433#14 0x000000010013bd90 in vm_call_method (th=0x7fff5fbfef60,cfp=0x7fff5fbfef60, num=8766224, blockptr=0x7fff5fbfef60,flag=140734799802208, id=4298129040, me=0x10038ec90, recv=4303733520) atvm_insnhelper.c:566...

36. Call three methods 10,000 times user system total realPre-delegate ruby(5919,0x7fff732a5180) malloc: *** error for object0x7fb18f80d750: incorrect checksum for freed object - object was probablymodified after being freed.*** set a breakpoint in malloc_error_break to debug

37. > gdb ./miniruby...(gdb) break malloc_error_breakFunction "malloc_error_break" not defined.Make breakpoint pending on future shared library load? (y or [n]) yBreakpoint 1 (malloc_error_break) pending.(gdb) run ./bug.rbStarting program: /Users/jballanc/Source/ruby/miniruby ./bug.rbReading symbols for shared libraries ++............................ doneBreakpoint 1 at 0x7fff8e607558Pending breakpoint 1 - "malloc_error_break" resolvedProgram received signal EXC_BAD_ACCESS, Could not access memory.Reason: KERN_INVALID_ADDRESS at address: 0x000000110060c9080x00007fff8e611cfb in tiny_free_list_remove_ptr ()(gdb)

41. Advanced Weaponry MallocStackLoggingNoCompact• man malloc• Set environment variables to log the stack during allocations• Run in GDB, use malloc_history to view stacks• CFLAGS=”-O0 -g” ./configure

42. (gdb) set environment MallocStackLoggingNoCompact=true(gdb) run ./bug.rbStarting program: /Users/jballanc/Source/ruby/miniruby ./bug.rbzsh(49630) malloc: recording malloc stacks to disk using standard recorderzsh(49630) malloc: stack logging compaction turned off; size of log files ondisk can increase rapidly...Reading symbols for shared libraries ++............................ doneminiruby(49630) malloc: recording malloc stacks to disk using standard recorderminiruby(49630) malloc: stack logging compaction turned off; size of log fileson disk can increase rapidlyminiruby(49630) malloc: stack logs deleted from /tmp/stack-logs.49630.arch.2JF4tB.indexminiruby(49630) malloc: stack logs being written into /tmp/stack-logs.49630.miniruby.lZtwm8.indexProgram received signal EXC_BAD_ACCESS, Could not access memory.Reason: KERN_INVALID_ADDRESS at address: 0x00000010000000080x00007fff8e611cfb in tiny_free_list_remove_ptr ()

46. (gdb) bt#0 0x00007fff8e611cfb in tiny_free_list_remove_ptr ()#1 0x00007fff8e60e76d in szone_free_definite_size ()#2 0x00007fff8e608898 in free ()#3 0x000000010006f6cc in vm_xfree (objspace=0x10081c400, ptr=0x1004918b0)at gc.c:830#4 0x000000010006fa76 in ruby_xfree (x=0x1004918b0) at gc.c:894...

51. BFGmalloclibgmalloc.dylib -- (Guard Malloc), an aggressive debugging malloc library

52. BFGmallocmallocgmalloc

59. (gdb) set environment MallocStackLoggingNoCompact=true(gdb) set environment DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib(gdb) run ./bug.rbStarting program: /Users/jballanc/Source/ruby/miniruby ./bug.rbGuardMalloc[zsh-50147]: recording malloc stacks to disk using standard recorderGuardMalloc[zsh-50147]: stack logging compaction turned off; size of log fileson disk can increase rapidly...

62. Program received signal EXC_BAD_ACCESS, Could not access memory.Reason: KERN_INVALID_ADDRESS at address: 0x0000000107203fe40x0000000100208b3c in rb_thread_mark (ptr=0x10085fce0) at vm.c:17331733! ! ! if (cfp->me) ((rb_method_entry_t *)cfp->me)->mark = 1;(gdb) p cfp->me$1 = (rb_method_entry_t *) 0x107203fe0

63. Program received signal EXC_BAD_ACCESS, Could not access memory.Reason: KERN_INVALID_ADDRESS at address: 0x0000000107203fe40x0000000100208b3c in rb_thread_mark (ptr=0x10085fce0) at vm.c:17331733! ! ! if (cfp->me) ((rb_method_entry_t *)cfp->me)->mark = 1;(gdb) p cfp->me$1 = (rb_method_entry_t *) 0x107203fe0

66. https://bugs.ruby-lang.org/issues/6171

67. Questions? Joshua Ballanco @manhattanmetrichttps://github.com/jballanc