Fluentdがよくわからなかった話

9,988 views

Published on

5分で全然終わらなかったのでスライドを上げておきました

Published in: Technology, Business

Fluentdがよくわからなかった話

  1. 1. fluentdの設定がよくわからない話@harukasanSunday, June 2, 13
  2. 2. harukasan・高専カンファレンス・久留米高専/九工大/筑波大・pixivのインフラ14ヶ月目Sunday, June 2, 13
  3. 3. Sunday, June 2, 13
  4. 4. Sunday, June 2, 13
  5. 5. Ruby on RailsSunday, June 2, 13
  6. 6. Ruby on RailsCapistranoSunday, June 2, 13
  7. 7. Ruby on RailsCapistranoFluentdSunday, June 2, 13
  8. 8. fluentdSunday, June 2, 13
  9. 9. Front AccessApp. AccessActivity - Sign-up, Sign-in, bookmark, ...PHP/Rails errorJS(client-side) errorMySQL slow queriesSolr slow requestsIRCSunday, June 2, 13
  10. 10. cappedmongodbAPP/DBin_tailextended in_tailout_mongoout_file_alternativeJSONAdmin viewminutely/dailybatchIRCGoogleDocsMySQLSunday, June 2, 13
  11. 11. fluentdSunday, June 2, 13
  12. 12. fluentdがよくわからなかった話Sunday, June 2, 13
  13. 13. Input PluginBuffer PluginOutput PluginInput data(tag, data)Output dataFluentd Basic architectureSunday, June 2, 13
  14. 14. ConfigureSunday, June 2, 13
  15. 15. <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
  16. 16. なんか動くSunday, June 2, 13
  17. 17. <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
  18. 18. <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
  19. 19. ようわからんSunday, June 2, 13
  20. 20. Sunday, June 2, 13
  21. 21. ようわからんSunday, June 2, 13
  22. 22. 質問Sunday, June 2, 13
  23. 23. 難易度英語 < RubySunday, June 2, 13
  24. 24. 難易度Ruby < 英語Sunday, June 2, 13
  25. 25. git clonegit@github.com:fluent/fluentd.gitSunday, June 2, 13
  26. 26. Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
  27. 27. Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
  28. 28. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    key  =  key.to_s    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit        chain.next        top  <<  data        return  false    elsif  @queue.size  >=  @buffer_queue_limit        raise  BufferQueueLimitError,                    "queue  size  exceeds  limit"    end    nc  =  new_chunk(key)Sunday, June 2, 13
  29. 29. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]Sunday, June 2, 13
  30. 30. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)Sunday, June 2, 13
  31. 31. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)←chunkSunday, June 2, 13
  32. 32. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)data←chunkSunday, June 2, 13
  33. 33. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)dataemit(key,data)←chunkSunday, June 2, 13
  34. 34. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)datadataemit(key,data)←chunkSunday, June 2, 13
  35. 35. Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)datadataemit(key,data)emit(key,data)←chunkSunday, June 2, 13
  36. 36. Fluent::BasicBuffer#emit    end    nc  =  new_chunk(key)    ok  =  false    begin        nc  <<  data        chain.next        flush_trigger  =  falsetop=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncSunday, June 2, 13
  37. 37. Fluent::BasicBuffer#emit    end    nc  =  new_chunk(key)    ok  =  false    begin        nc  <<  data        chain.next        flush_trigger  =  falsetop=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdataSunday, June 2, 13
  38. 38. Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdataSunday, June 2, 13
  39. 39. Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdata@queueSunday, June 2, 13
  40. 40. Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdata@queueSunday, June 2, 13
  41. 41. Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data)data@queueSunday, June 2, 13
  42. 42. Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data)data@queueSunday, June 2, 13
  43. 43. Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
  44. 44. Fluent::BasicBuffer#pushdef  push(key)    top  =  @map[key]    if  !top  ||  top.empty?        return  false    end    @queue.synchronize  do        enqueue(top)        @queue  <<  top        @map.delete(key)    end    return  trueendSunday, June 2, 13
  45. 45. Fluent::BasicBuffer#push    @queue.synchronize  do        enqueue(top)        @queue  <<  top        @map.delete(key)    end    return  trueendtop=@map[key]datadatadata@queueSunday, June 2, 13
  46. 46. Fluent::BasicBuffer#push    @queue.synchronize  do        enqueue(top)        @queue  <<  top        @map.delete(key)    end    return  trueenddatadatadata@queueSunday, June 2, 13
  47. 47. Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
  48. 48. Fluent::BasicBuffer#popdef  pop(out)    chunk  =  @queue.first    if  !chunk.empty?        write_chunk(chunk,  out)    end    @queue.delete_if  {|c|        c.object_id  ==  chunk.object_id    }    chunk.purge    return  !@queue.empty?endSunday, June 2, 13
  49. 49. Fluent::BasicBuffer#popdatadatadata@queuedef  pop(out)    chunk  =  @queue.first    if  !chunk.empty?        write_chunk(chunk,  out)    end    @queue.delete_if  {|c|  Sunday, June 2, 13
  50. 50. Fluent::BasicBuffer#popdatadatadata@queuedef  pop(out)    chunk  =  @queue.first    if  !chunk.empty?        write_chunk(chunk,  out)    end    @queue.delete_if  {|c|  Sunday, June 2, 13
  51. 51. Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(chunk)BufferInputsometing(chunk)pushpopSunday, June 2, 13
  52. 52. Configure parametersSunday, June 2, 13
  53. 53. <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
  54. 54. buffer_chunk_limitbuffer chunkのサイズ制限Sunday, June 2, 13
  55. 55. buffer_queue_limitqueueにはいるchunkの数Sunday, June 2, 13
  56. 56. flush_intervalqueueがpopされる間隔Sunday, June 2, 13
  57. 57. まとめSunday, June 2, 13
  58. 58. documentを書くのは面倒なのでdocumentを読みたい人はdocumentを書こうdocumentがなくてもcodeが小さければ読めるよくわからなかったらcodeを読もうSunday, June 2, 13

×