GCNO effort CLEAN memory
GCNO effort CLEAN memory        SOME
Who am I?       Alex NaumchukSoftware Architect at     Consulter at Brightgrove   alex.naumchuk@gmail.com
What IS GC?
What IS GC?Developers
80k req/min     10k db queries/sec      125 prod servers              avg response time: 25ms
Automatic collector reclaiming garbage               C, C++
Automatic collector reclaiming garbage               C, C++           Ruby, Java, .NET
Automatic collector reclaiming garbage               C, C++          ALLY                          MA NU                 D...
Automatic collector reclaiming garbage               C, C++          ALLY                          MA NU                 D...
Should you care about memory?
Should you care about memory?           YES!
Should you care about memory?             YES!       at least know how it works
REE/MRI Memory organizationHeaps        slot slot slot slot slot        slot slot slot slot slot        slot slot slot slo...
Slot RVALUEtypedef struct RVALUE {    union {  struct {      unsigned long flags;   /* always 0 for freed obj */      stru...
REE GC Options
Default SettingsRUBY_HEAP_SLOTS                  10000
Default SettingsRUBY_HEAP_SLOTS                  10000RUBY_HEAP_SLOTS_INCREMENT        10000
Default SettingsRUBY_HEAP_SLOTS                  10000RUBY_HEAP_SLOTS_INCREMENT        10000RUBY_HEAP_SLOTS_GROWTH_FACTOR ...
Default SettingsRUBY_HEAP_SLOTS                    10000RUBY_HEAP_SLOTS_INCREMENT         10000RUBY_HEAP_SLOTS_GROWTH_FACT...
Default SettingsRUBY_HEAP_SLOTS                    10000RUBY_HEAP_SLOTS_INCREMENT         10000RUBY_HEAP_SLOTS_GROWTH_FACT...
Ideeli Settings                                   MM       FERUBY_HEAP_SLOTS                 6000000   2400000RUBY_HEAP_SL...
When GC runs      malloc limit exceeded    min free slots < free slotsnot enough slots to allocate value
Mark Phase
GC Flow  Mark Sweep Finalize
Statistics                 >>ObjectSpace.statisticsNumber of objects : 1631452 (1291302 AST nodes, 79.15%)Heap slot size  ...
Other StatsGC.enable_statsGC.disable_statsGC.collectionsGC.timeGC.growthGC.clear_statsGC.dumpGC.allocated_sizeGC.num_alloc...
Let’s Play Let’s play
CodeGC.enable_statsrequire ‘config/environment’GC.timeGC.num_allocationsGC.collections
Default optionsGC.enable_statsrequire ‘config/environment’GC.time                     ~2 SecondsGC.num_allocations         ...
ObjectSpace.statisticsNumber of objects : 1643029 (1265158 AST nodes, 77.00%)Heap slot size        : 40GC cycles so far   ...
Default options• Too many GC cycles• Many heaps• Bad heap utilization
New optionsRUBY_HEAP_MIN_SLOTS = 1_800_000RUBY_GC_MALLOC_LIMIT = 50_000_000  GC.time              ~1 Second  GC.num_alloca...
New options    RUBY_HEAP_MIN_SLOTS = 1_800_000    RUBY_GC_MALLOC_LIMIT = 50_000_000Number of objects : 1682325 (1272351 AS...
Can we do it even better?
Can we do it even better?        YES!
Enhanced optionsRUBY_HEAP_MIN_SLOTS = 2_100_000RUBY_GC_MALLOC_LIMIT = 100_000_000   GC.time              ~0.8 Second   GC....
Enhanced options   RUBY_HEAP_MIN_SLOTS = 2_100_000   RUBY_GC_MALLOC_LIMIT = 100_000_000Number of objects : 1816515 (129727...
What about memory? Before 325.9M After 347.2M         21.3M
Garbage Recon
Garbage Recon                                    AVG GC   AVG Object     AVG Req              Action                      ...
In between request GC      User Request       Response          GC
It’s a trap
The Codeclass Test  attr_accessor :i_val  def initialize    @i_val = []    300_000.times do |i|      @i_val << "ruby-#{i}"...
The Testtest = Test.newGC.startt.i_val = nilt.say_helloGC.startt = []GC.start
Resultstest = Test.new # 1504032t.i_val = nil   # 3922t = []          # 3916
Conclusion• be careful with the big sets of data• don’t store big amount of data in object with long life time• even if cu...
Questions?
Upcoming SlideShare
Loading in …5
×

Use Ruby GC in full..

2,664 views
2,552 views

Published on

Alexander Naumchuk - «GC time garbage collection» (Ideeli)

2 Comments
1 Like
Statistics
Notes
No Downloads
Views
Total views
2,664
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
2
Likes
1
Embeds 0
No embeds

No notes for slide
  • \n\n
  • \n\n
  • \n
  • we are dealing with gc every day, saw that brown piece in newrelic?\n
  • Is that that brow bar at the bottom in newrelic?\n
  • GC is automtic collector that reclaiming garbage or memory occupied by objects.\n\nIn some languages you manage memory manually: C, C++\n You have to free all allocations, you have to avoid memory leaks, you have deal with double free problems, dangling pointer bugs. And tons of other problems that give you permanent headache.\n\nBut we are cool dynamic devs and do not care about that.\n\n \n
  • GC is automtic collector that reclaiming garbage or memory occupied by objects.\n\nIn some languages you manage memory manually: C, C++\n You have to free all allocations, you have to avoid memory leaks, you have deal with double free problems, dangling pointer bugs. And tons of other problems that give you permanent headache.\n\nBut we are cool dynamic devs and do not care about that.\n\n \n
  • GC is automtic collector that reclaiming garbage or memory occupied by objects.\n\nIn some languages you manage memory manually: C, C++\n You have to free all allocations, you have to avoid memory leaks, you have deal with double free problems, dangling pointer bugs. And tons of other problems that give you permanent headache.\n\nBut we are cool dynamic devs and do not care about that.\n\n \n
  • But should we care about it?\nAnswer is yes - we should\nWe are not managing memory manually but we should know how it work.\n\n
  • But should we care about it?\nAnswer is yes - we should\nWe are not managing memory manually but we should know how it work.\n\n
  • All ruby objects are stored in heaps. Each heap have number of slots within it.\n\n
  • Each slot of a heap hold RVALUE structure.\nIt&amp;#x2019;s a union of all possible ruby data objects. Even source node tree is stored in heap.\nBut that structure is not representing actual data storage. Like object references i_val table that is reference to different RVALUE objects.\n\n
  • \n
  • H_SLOTS - Number of initial slots in heap\nH_SLOTS_INC - Number of additional heap slots\nGROWTH_FACTOR - Multiplier to increase heap slots size\nG_MALLOC_LIMIT - limit of allocations before GC, literally how many times malloc has been called\nFREE_MIN - NUMBER of \n
  • H_SLOTS - Number of initial slots in heap\nH_SLOTS_INC - Number of additional heap slots\nGROWTH_FACTOR - Multiplier to increase heap slots size\nG_MALLOC_LIMIT - limit of allocations before GC, literally how many times malloc has been called\nFREE_MIN - NUMBER of \n
  • H_SLOTS - Number of initial slots in heap\nH_SLOTS_INC - Number of additional heap slots\nGROWTH_FACTOR - Multiplier to increase heap slots size\nG_MALLOC_LIMIT - limit of allocations before GC, literally how many times malloc has been called\nFREE_MIN - NUMBER of \n
  • H_SLOTS - Number of initial slots in heap\nH_SLOTS_INC - Number of additional heap slots\nGROWTH_FACTOR - Multiplier to increase heap slots size\nG_MALLOC_LIMIT - limit of allocations before GC, literally how many times malloc has been called\nFREE_MIN - NUMBER of \n
  • H_SLOTS - Number of initial slots in heap\nH_SLOTS_INC - Number of additional heap slots\nGROWTH_FACTOR - Multiplier to increase heap slots size\nG_MALLOC_LIMIT - limit of allocations before GC, literally how many times malloc has been called\nFREE_MIN - NUMBER of \n
  • The ideeli settings were chosen ob node purpose and average stat for that kind of node\n
  • Every object creation checks available allocations\nGC can run at any moment\n
  • At mark phase interpreter looks for a current variables references. It&amp;#x2019;s not ref based, it&amp;#x2019;s scope and lookup based.\n
  • In sweep phase any allocations are prohibited\n\n
  • REE specific. Stats are collected across all process life time frame so the GC cycles increasing\nAst nodes is clas varmap scope singelton etc. Terminal objects are non-object types (regex, file, float, bignum, node)\n
  • The last couple one do not require gc stats to be enabled.\nNum allocations is the num of objects that were allocated since the script started\n
  • \n
  • Simple code that load&amp;#x2019;s our app env in dev mode\n
  • Spent 2 sec in GC, allocated 5.5m objects and made 31!!!! GC cycles\n
  • \n
  • Have 31 GC cycle, 9 heaps and 46% unused almost half of it\n
  • increase min slots to fit app into one heap\n
  • Much more better utilization, only 6% unused, but it might be dangerous as well because of it&amp;#x2019;s only env loaded.\n
  • \n
  • Give some more space for heap, and increase malloc limit.\n
  • As a result decent heap utilization, low number of gc cycles\n
  • Memory consumption increased by 21 Mb.\n
  • Simple middleware the gathering GC stats\n
  • Sample stats got for mm node\n
  • Unicorn supports in between req gc. It&amp;#x2019;s made in between of user requests. That&amp;#x2019;s simple middleware that can be controlled. Gotcha is that wrong params may lead to significant throughput drop. \nGC.start - forces garbage collection weather or not it&amp;#x2019;s required\n
  • \n
  • \n
  • \n
  • any of 2 last lines would have the same effect - dramatically decreasing live objects number\nWithout explicitly setting instance variable - the whole object set would live until object is in scope\n\n
  • Every object creation checks available allocations\n\n
  • \n
  • Use Ruby GC in full..

    1. 1. GCNO effort CLEAN memory
    2. 2. GCNO effort CLEAN memory SOME
    3. 3. Who am I? Alex NaumchukSoftware Architect at Consulter at Brightgrove alex.naumchuk@gmail.com
    4. 4. What IS GC?
    5. 5. What IS GC?Developers
    6. 6. 80k req/min 10k db queries/sec 125 prod servers avg response time: 25ms
    7. 7. Automatic collector reclaiming garbage C, C++
    8. 8. Automatic collector reclaiming garbage C, C++ Ruby, Java, .NET
    9. 9. Automatic collector reclaiming garbage C, C++ ALLY MA NU D o it Ruby, Java, .NET
    10. 10. Automatic collector reclaiming garbage C, C++ ALLY MA NU D o it Ruby, Java, .NET about it ’t C ARE Don
    11. 11. Should you care about memory?
    12. 12. Should you care about memory? YES!
    13. 13. Should you care about memory? YES! at least know how it works
    14. 14. REE/MRI Memory organizationHeaps slot slot slot slot slot slot slot slot slot slot slot slot slot slot slot
    15. 15. Slot RVALUEtypedef struct RVALUE { union { struct { unsigned long flags; /* always 0 for freed obj */ struct RVALUE *next; } free; struct RBasic basic; struct RObject object; struct RClass klass; struct RFloat flonum; struct RString string; struct RArray array; struct RRegexp regexp; struct RHash hash; struct RData data; struct RStruct rstruct; struct RBignum bignum; struct RFile file; struct RNode node; struct RMatch match; struct RVarmap varmap; struct SCOPE scope; } as;} RVALUE;
    16. 16. REE GC Options
    17. 17. Default SettingsRUBY_HEAP_SLOTS 10000
    18. 18. Default SettingsRUBY_HEAP_SLOTS 10000RUBY_HEAP_SLOTS_INCREMENT 10000
    19. 19. Default SettingsRUBY_HEAP_SLOTS 10000RUBY_HEAP_SLOTS_INCREMENT 10000RUBY_HEAP_SLOTS_GROWTH_FACTOR 1.8
    20. 20. Default SettingsRUBY_HEAP_SLOTS 10000RUBY_HEAP_SLOTS_INCREMENT 10000RUBY_HEAP_SLOTS_GROWTH_FACTOR 1.8RUBY_GC_MALLOC_LIMIT 8000000
    21. 21. Default SettingsRUBY_HEAP_SLOTS 10000RUBY_HEAP_SLOTS_INCREMENT 10000RUBY_HEAP_SLOTS_GROWTH_FACTOR 1.8RUBY_GC_MALLOC_LIMIT 8000000RUBY_HEAP_FREE_MIN 4096
    22. 22. Ideeli Settings MM FERUBY_HEAP_SLOTS 6000000 2400000RUBY_HEAP_SLOTS_INCREMENT 100000 100000RUBY_HEAP_SLOTS_GROWTH_FACTOR 1 1RUBY_GC_MALLOC_LIMIT 100000000 40000000RUBY_HEAP_FREE_MIN 3500000 100000
    23. 23. When GC runs malloc limit exceeded min free slots < free slotsnot enough slots to allocate value
    24. 24. Mark Phase
    25. 25. GC Flow Mark Sweep Finalize
    26. 26. Statistics >>ObjectSpace.statisticsNumber of objects : 1631452 (1291302 AST nodes, 79.15%)Heap slot size : 40GC cycles so far : 32Number of heaps : 9Total size of objects : 63728.59 KBTotal size of heaps : 119579.81 KB (55851.22 KB = 46.71% unused)Leading free slots : 1154340 (45091.41 KB = 37.71%)Trailing free slots : 4 (0.16 KB = 0.00%)Number of contiguous groups of 16 slots: 76771 (40.13%)Number of terminal objects: 8973 (0.29%)
    27. 27. Other StatsGC.enable_statsGC.disable_statsGC.collectionsGC.timeGC.growthGC.clear_statsGC.dumpGC.allocated_sizeGC.num_allocationsObjectSpace.live_objectsObjectSpace.allocated_objects
    28. 28. Let’s Play Let’s play
    29. 29. CodeGC.enable_statsrequire ‘config/environment’GC.timeGC.num_allocationsGC.collections
    30. 30. Default optionsGC.enable_statsrequire ‘config/environment’GC.time ~2 SecondsGC.num_allocations 5_635_070GC.collections 31
    31. 31. ObjectSpace.statisticsNumber of objects : 1643029 (1265158 AST nodes, 77.00%)Heap slot size : 40GC cycles so far : 31Number of heaps : 9Total size of objects : 64180.82 KBTotal size of heaps : 119579.42 KB (55398.60 KB = 46.33% unused)Leading free slots : 1093423 (42711.84 KB = 35.72%)Trailing free slots : 0 (0.00 KB = 0.00%)Number of contiguous groups of 16 slots: 73754 (38.55%)Number of terminal objects: 7274 (0.24%)
    32. 32. Default options• Too many GC cycles• Many heaps• Bad heap utilization
    33. 33. New optionsRUBY_HEAP_MIN_SLOTS = 1_800_000RUBY_GC_MALLOC_LIMIT = 50_000_000 GC.time ~1 Second GC.num_allocations 5_634_856 GC.collections 6
    34. 34. New options RUBY_HEAP_MIN_SLOTS = 1_800_000 RUBY_GC_MALLOC_LIMIT = 50_000_000Number of objects : 1682325 (1272351 AST nodes, 75.63%)Heap slot size : 40GC cycles so far :6Number of heaps : 1Total size of objects : 65715.82 KBTotal size of heaps : 70312.51 KB (4596.69 KB = 6.54% unused)Leading free slots : 39246 (1533.05 KB = 2.18%)Trailing free slots : 0 (0.00 KB = 0.00%)Number of contiguous groups of 16 slots: 4305 (3.83%)Number of terminal objects: 7435 (0.41%)
    35. 35. Can we do it even better?
    36. 36. Can we do it even better? YES!
    37. 37. Enhanced optionsRUBY_HEAP_MIN_SLOTS = 2_100_000RUBY_GC_MALLOC_LIMIT = 100_000_000 GC.time ~0.8 Second GC.num_allocations 5_634_892 GC.collections 3
    38. 38. Enhanced options RUBY_HEAP_MIN_SLOTS = 2_100_000 RUBY_GC_MALLOC_LIMIT = 100_000_000Number of objects : 1816515 (1297270 AST nodes, 71.42%)Heap slot size : 40GC cycles so far :3Number of heaps : 1Total size of objects : 70957.62 KBTotal size of heaps : 82031.26 KB (11073.64 KB = 13.50% unused)Leading free slots : 0 (0.00 KB = 0.00%)Trailing free slots : 0 (0.00 KB = 0.00%)Number of contiguous groups of 16 slots: 5807 (4.42%)Number of terminal objects: 7969 (0.38%)
    39. 39. What about memory? Before 325.9M After 347.2M 21.3M
    40. 40. Garbage Recon
    41. 41. Garbage Recon AVG GC AVG Object AVG Req Action Rounds Allocations between GC /cart_items/index 0.61 330651 2 /cart_items/ID/ 1.21 651679 1 /cart_items 0.93 498881 1/store/confirm_reservation_address 0.15 76084 6 /store/confirm_reservation 1.37 724095 1
    42. 42. In between request GC User Request Response GC
    43. 43. It’s a trap
    44. 44. The Codeclass Test attr_accessor :i_val def initialize @i_val = [] 300_000.times do |i| @i_val << "ruby-#{i}" * 10 end end def say_hello "hello" endend
    45. 45. The Testtest = Test.newGC.startt.i_val = nilt.say_helloGC.startt = []GC.start
    46. 46. Resultstest = Test.new # 1504032t.i_val = nil # 3922t = [] # 3916
    47. 47. Conclusion• be careful with the big sets of data• don’t store big amount of data in object with long life time• even if current request wont spend much time in GC the next request would do it• the more objects - the longer is mark phase• symbols are not collected but increase mark phase time
    48. 48. Questions?

    ×