Successfully reported this slideshow.

Fluentdでログ収集「だけ」やる話 #study2study

5,249 views

Published on

Webサーバ勉強会 2012/03/29

Published in: Technology
  • Be the first to comment

Fluentdでログ収集「だけ」やる話 #study2study

  1. 1. Fluentdでログ収集「だけ」やる話 What to do is to collect raw logs #study2study 2012/03/29 @tagomoris12年3月30日金曜日
  2. 2. @tagomoris NHN Japan Corp. Web Service Business Division Development Department 2 Development Platforms12年3月30日金曜日
  3. 3. Fluentd • That we all love.12年3月30日金曜日
  4. 4. Basic method • Fluentd in_tail + out/in_forward + out_file Web server fluentd Collect server fluentd Web server fluentd Web server fluentd12年3月30日金曜日
  5. 5. in_tail + out_file: JSON 20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031", 20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031", 20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031", • JSON is good for new systems • But traditional (historically important) systems requires raw logs • How do we get raw logs with Fluentd ?12年3月30日金曜日
  6. 6. we neeeeeed raw log • fluent-agent-lite • Fluentd in_forward + out_file_alternative Web server fluent-agent-lite Collect server fluentd Web server fluent-agent-lite Web server fluent-agent-lite12年3月30日金曜日
  7. 7. fluent-agent-lite • https://github.com/tagomoris/fluent-agent-lite • easy to intall, run • run with perl 5.8 (only!) • bin/install.sh or rpm (.spec bundled) • start-stop script bundled • lightweight, raw log intensive • read a line, send it as one msg with one field • with high throughput and low load12年3月30日金曜日
  8. 8. fluent-agent-lite • many features • fluentd cluster support (select one randomly) • failover support (primary / secondary servers) • configurations for high performance12年3月30日金曜日
  9. 9. fluent-plugin-file-alternative • 100% compatible to standard out_file • with same configuration, get same output • simple configuration • path w/ time placeholder w/o time_slice_format • /path/to/log/access.%Y-%m-%d.log12年3月30日金曜日
  10. 10. fluent-plugin-file-alternative • custom output line formats • time of fluentd message OR NOT • tag of fluentd message OR NOT • custom output data formats • whole record as json • selected single field • selected multi fields seperated by TAB/SPACE/ COMMA12年3月30日金曜日
  11. 11. fluent-plugin-file-alternative • Misc • add newline at the end of data OR NOT • appending data to single file with flush_interval (without compression)12年3月30日金曜日
  12. 12. simple configuration <match fluentd.status.gc.**> type file path /var/log/fluentd.status/gc time_slice_format %Y-%m-%d.%H </match> <match fluentd.status.gc.**> type file_alternative path /var/log/fluentd.status/gc.%Y-%m-%d.%H.log </match>12年3月30日金曜日
  13. 13. for raw logs <match apache.log.**> type file_alternative path /var/log/archive/access.%Y-%m-%d.%H.log output_include_time false output_include_tag false output_data_type attr:message add_newline true </match>12年3月30日金曜日
  14. 14. Thanks!12年3月30日金曜日

×