• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Fluentdがよくわからなかった話
 

Fluentdがよくわからなかった話

on

  • 9,868 views

5分で全然終わらなかったのでスライドを上げておきました

5分で全然終わらなかったのでスライドを上げておきました

Statistics

Views

Total Views
9,868
Views on SlideShare
9,707
Embed Views
161

Actions

Likes
37
Downloads
0
Comments
0

4 Embeds 161

https://twitter.com 152
http://tweetedtimes.com 6
http://localhost 2
https://web.tweetdeck.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Fluentdがよくわからなかった話 Fluentdがよくわからなかった話 Presentation Transcript

    • fluentdの設定がよくわからない話@harukasanSunday, June 2, 13
    • harukasan・高専カンファレンス・久留米高専/九工大/筑波大・pixivのインフラ14ヶ月目Sunday, June 2, 13
    • Sunday, June 2, 13
    • Sunday, June 2, 13
    • Ruby on RailsSunday, June 2, 13
    • Ruby on RailsCapistranoSunday, June 2, 13
    • Ruby on RailsCapistranoFluentdSunday, June 2, 13
    • fluentdSunday, June 2, 13
    • Front AccessApp. AccessActivity - Sign-up, Sign-in, bookmark, ...PHP/Rails errorJS(client-side) errorMySQL slow queriesSolr slow requestsIRCSunday, June 2, 13
    • cappedmongodbAPP/DBin_tailextended in_tailout_mongoout_file_alternativeJSONAdmin viewminutely/dailybatchIRCGoogleDocsMySQLSunday, June 2, 13
    • fluentdSunday, June 2, 13
    • fluentdがよくわからなかった話Sunday, June 2, 13
    • Input PluginBuffer PluginOutput PluginInput data(tag, data)Output dataFluentd Basic architectureSunday, June 2, 13
    • ConfigureSunday, June 2, 13
    • <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
    • なんか動くSunday, June 2, 13
    • <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
    • <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
    • ようわからんSunday, June 2, 13
    • Sunday, June 2, 13
    • ようわからんSunday, June 2, 13
    • 質問Sunday, June 2, 13
    • 難易度英語 < RubySunday, June 2, 13
    • 難易度Ruby < 英語Sunday, June 2, 13
    • git clonegit@github.com:fluent/fluentd.gitSunday, June 2, 13
    • Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
    • Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    key  =  key.to_s    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit        chain.next        top  <<  data        return  false    elsif  @queue.size  >=  @buffer_queue_limit        raise  BufferQueueLimitError,                    "queue  size  exceeds  limit"    end    nc  =  new_chunk(key)Sunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]Sunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)Sunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)←chunkSunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)data←chunkSunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)dataemit(key,data)←chunkSunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)datadataemit(key,data)←chunkSunday, June 2, 13
    • Fluent::BasicBuffer#emitdef  emit(key,  data,  chain)    ...    top  =  (@map[key]  ||=  new_chunk(key))    if  top.size  +  data.bytesize  <=  @buffer_chunk_limit...        top  <<  data        return  falsetop=@map[key]emit(key,data)datadataemit(key,data)emit(key,data)←chunkSunday, June 2, 13
    • Fluent::BasicBuffer#emit    end    nc  =  new_chunk(key)    ok  =  false    begin        nc  <<  data        chain.next        flush_trigger  =  falsetop=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncSunday, June 2, 13
    • Fluent::BasicBuffer#emit    end    nc  =  new_chunk(key)    ok  =  false    begin        nc  <<  data        chain.next        flush_trigger  =  falsetop=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdataSunday, June 2, 13
    • Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdataSunday, June 2, 13
    • Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdata@queueSunday, June 2, 13
    • Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data) ncdata@queueSunday, June 2, 13
    • Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data)data@queueSunday, June 2, 13
    • Fluent::BasicBuffer#emit        flush_trigger  =  false        @queue.synchronize  {            enqueue(top)            flush_trigger  =  @queue.empty?            @queue  <<  top            @map[key]  =  nc        }top=@map[key]emit(key,data)datadataemit(key,data)emit(key,data)data@queueSunday, June 2, 13
    • Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
    • Fluent::BasicBuffer#pushdef  push(key)    top  =  @map[key]    if  !top  ||  top.empty?        return  false    end    @queue.synchronize  do        enqueue(top)        @queue  <<  top        @map.delete(key)    end    return  trueendSunday, June 2, 13
    • Fluent::BasicBuffer#push    @queue.synchronize  do        enqueue(top)        @queue  <<  top        @map.delete(key)    end    return  trueendtop=@map[key]datadatadata@queueSunday, June 2, 13
    • Fluent::BasicBuffer#push    @queue.synchronize  do        enqueue(top)        @queue  <<  top        @map.delete(key)    end    return  trueenddatadatadata@queueSunday, June 2, 13
    • Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(data)BufferInputsometing(data)pushpopSunday, June 2, 13
    • Fluent::BasicBuffer#popdef  pop(out)    chunk  =  @queue.first    if  !chunk.empty?        write_chunk(chunk,  out)    end    @queue.delete_if  {|c|        c.object_id  ==  chunk.object_id    }    chunk.purge    return  !@queue.empty?endSunday, June 2, 13
    • Fluent::BasicBuffer#popdatadatadata@queuedef  pop(out)    chunk  =  @queue.first    if  !chunk.empty?        write_chunk(chunk,  out)    end    @queue.delete_if  {|c|  Sunday, June 2, 13
    • Fluent::BasicBuffer#popdatadatadata@queuedef  pop(out)    chunk  =  @queue.first    if  !chunk.empty?        write_chunk(chunk,  out)    end    @queue.delete_if  {|c|  Sunday, June 2, 13
    • Fluent::BufferedOutputemit(tag,data)emit(tag,data)BufferedOutputwrite(chunk)BufferInputsometing(chunk)pushpopSunday, June 2, 13
    • Configure parametersSunday, June 2, 13
    • <source>type tailpath /tmp/test.logtag test.logformat json</source><match test.log>type mongohost localhostdatabase testcollection logbuffer_chunk_limit 2mbuffer_queue_limit 256flush_interval 1s</match>Sunday, June 2, 13
    • buffer_chunk_limitbuffer chunkのサイズ制限Sunday, June 2, 13
    • buffer_queue_limitqueueにはいるchunkの数Sunday, June 2, 13
    • flush_intervalqueueがpopされる間隔Sunday, June 2, 13
    • まとめSunday, June 2, 13
    • documentを書くのは面倒なのでdocumentを読みたい人はdocumentを書こうdocumentがなくてもcodeが小さければ読めるよくわからなかったらcodeを読もうSunday, June 2, 13