Webサーバ勉強会 2012/03/29

1. Fluentdでログ収集「だけ」やる話
What to do is to collect raw logs
#study2study 2012/03/29
@tagomoris
12年3月30日金曜日

2. @tagomoris
NHN Japan Corp.
Web Service Business Division
Development Department 2
Development Platforms
12年3月30日金曜日

3. Fluentd
• 'That' we all love.
12年3月30日金曜日

4. Basic method
• Fluentd in_tail + out/in_forward + out_file
Web server
fluentd
Collect server
fluentd
Web server
fluentd
Web server
fluentd
12年3月30日金曜日

5. in_tail + out_file: JSON
20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031",
20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031",
20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031",
• JSON is good for new systems
• But traditional (historically important)
systems requires raw logs
• How do we get raw logs with Fluentd ?
12年3月30日金曜日

6. we neeeeeed raw log
• fluent-agent-lite
• Fluentd in_forward + out_file_alternative
Web server
fluent-agent-lite
Collect server
fluentd
Web server
fluent-agent-lite
Web server
fluent-agent-lite
12年3月30日金曜日

7. fluent-agent-lite
• https://github.com/tagomoris/fluent-agent-lite
• easy to intall, run
• run with perl 5.8 (only!)
• bin/install.sh or rpm (.spec bundled)
• start-stop script bundled
• lightweight, raw log intensive
• read a line, send it as one msg with one field
• with high throughput and low load
12年3月30日金曜日

8. fluent-agent-lite
• many features
• fluentd cluster support (select one randomly)
• failover support (primary / secondary servers)
• configurations for high performance
12年3月30日金曜日

9. fluent-plugin-file-alternative
• 100% compatible to standard out_file
• with same configuration, get same output
• simple configuration
• path w/ time placeholder w/o
time_slice_format
• /path/to/log/access.%Y-%m-%d.log
12年3月30日金曜日

10. fluent-plugin-file-alternative
• custom output line formats
• time of fluentd message OR NOT
• tag of fluentd message OR NOT
• custom output data formats
• whole record as json
• selected single field
• selected multi fields seperated by TAB/SPACE/
COMMA
12年3月30日金曜日

11. fluent-plugin-file-alternative
• Misc
• add newline at the end of data OR NOT
• appending data to single file with
flush_interval (without compression)
12年3月30日金曜日

12. simple configuration
<match fluentd.status.gc.**>
type file
path /var/log/fluentd.status/gc
time_slice_format %Y-%m-%d.%H
</match>
<match fluentd.status.gc.**>
type file_alternative
path /var/log/fluentd.status/gc.%Y-%m-%d.%H.log
</match>
12年3月30日金曜日

13. for raw logs
<match apache.log.**>
type file_alternative
path /var/log/archive/access.%Y-%m-%d.%H.log
output_include_time false
output_include_tag false
output_data_type attr:message
add_newline true
</match>
12年3月30日金曜日

14. Thanks!
12年3月30日金曜日