The document discusses using Fluentd to collect raw logs. It describes how the basic method uses in_tail and out_file plugins, but that traditional systems require raw logs. The fluent-agent-lite and fluent-plugin-file-alternative plugins are introduced as ways to collect and output raw logs with Fluentd while maintaining high performance and throughput. Configuration examples are provided for using these plugins to output raw logs to files.

1. Fluentdでログ収集「だけ」やる話
What to do is to collect raw logs
#study2study 2012/03/29
@tagomoris
12年3月30日金曜日

2. @tagomoris
NHN Japan Corp.
Web Service Business Division
Development Department 2
Development Platforms
12年3月30日金曜日

3. Fluentd
• 'That' we all love.
12年3月30日金曜日

4. Basic method
• Fluentd in_tail + out/in_forward + out_file
Web server
fluentd
Collect server
fluentd
Web server
fluentd
Web server
fluentd
12年3月30日金曜日

5. in_tail + out_file: JSON
20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031",
20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031",
20120501 23:59:00 blog {"path":"/dankogai/archives/51780443.html","code":"200","size":"5031",
• JSON is good for new systems
• But traditional (historically important)
systems requires raw logs
• How do we get raw logs with Fluentd ?
12年3月30日金曜日

6. we neeeeeed raw log
• fluent-agent-lite
• Fluentd in_forward + out_file_alternative
Web server
fluent-agent-lite
Collect server
fluentd
Web server
fluent-agent-lite
Web server
fluent-agent-lite
12年3月30日金曜日

7. fluent-agent-lite
• https://github.com/tagomoris/fluent-agent-lite
• easy to intall, run
• run with perl 5.8 (only!)
• bin/install.sh or rpm (.spec bundled)
• start-stop script bundled
• lightweight, raw log intensive
• read a line, send it as one msg with one field
• with high throughput and low load
12年3月30日金曜日

8. fluent-agent-lite
• many features
• fluentd cluster support (select one randomly)
• failover support (primary / secondary servers)
• configurations for high performance
12年3月30日金曜日

9. fluent-plugin-file-alternative
• 100% compatible to standard out_file
• with same configuration, get same output
• simple configuration
• path w/ time placeholder w/o
time_slice_format
• /path/to/log/access.%Y-%m-%d.log
12年3月30日金曜日

10. fluent-plugin-file-alternative
• custom output line formats
• time of fluentd message OR NOT
• tag of fluentd message OR NOT
• custom output data formats
• whole record as json
• selected single field
• selected multi fields seperated by TAB/SPACE/
COMMA
12年3月30日金曜日

11. fluent-plugin-file-alternative
• Misc
• add newline at the end of data OR NOT
• appending data to single file with
flush_interval (without compression)
12年3月30日金曜日

12. simple configuration
<match fluentd.status.gc.**>
type file
path /var/log/fluentd.status/gc
time_slice_format %Y-%m-%d.%H
</match>
<match fluentd.status.gc.**>
type file_alternative
path /var/log/fluentd.status/gc.%Y-%m-%d.%H.log
</match>
12年3月30日金曜日

13. for raw logs
<match apache.log.**>
type file_alternative
path /var/log/archive/access.%Y-%m-%d.%H.log
output_include_time false
output_include_tag false
output_data_type attr:message
add_newline true
</match>
12年3月30日金曜日

14. Thanks!
12年3月30日金曜日