• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Fluetnd meetup japan #3 @studio3104
 

Fluetnd meetup japan #3 @studio3104

on

  • 3,212 views

 

Statistics

Views

Total Views
3,212
Views on SlideShare
1,821
Embed Views
1,391

Actions

Likes
8
Downloads
26
Comments
0

20 Embeds 1,391

http://d.hatena.ne.jp 594
http://typopress.blogspot.jp 469
http://studio3104.hatenablog.com 144
http://mfks17.hateblo.jp 75
http://hatenatunnel.appspot.com 33
http://typopress.blogspot.com 31
https://twitter.com 22
http://webcache.googleusercontent.com 5
http://typopress.blogspot.de 4
http://typopress.blogspot.tw 2
http://typopress.blogspot.kr 2
http://friendfeed.com 2
http://gimite.net 1
http://reader.livedoor.com 1
http://typopress.blogspot.fi 1
http://typopress.blogspot.ca 1
http://typopress.blogspot.co.uk 1
http://typopress.blogspot.co.nz 1
http://translate.googleusercontent.com 1
http://typopress.blogspot.in 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Fluetnd meetup japan #3 @studio3104 Fluetnd meetup japan #3 @studio3104 Presentation Transcript

    • Talk 4th Section at Fluentd Meetup #3Small Startup Fluentd For Fluentd Beginners @studio3104
    • @studio3104 (Satoshi SUZUKI)-- FluentdドリヴンでRubyはじめました-- WEB業界2年目-- 妻、娘、息子います-- ソシャゲ、公式サイト、パチ(スロ)ンコ-- こんなのやってます →→→→→→→-- Perlと心中するらしいです 自己紹介
    • td-agentfluent-agent-lite 今日お話すること fluent-agent-lite in_forward out_copy access log access log out_file_alternative Fluentd Design Pattern的な access log error log out_parser error log error log nginx out_copy out_exec_filterWEB SERVERS out_(mail|mongo|redis out_exec_filter |file|mysql|s3) out_rewrite 今日お話しないこと Mail / Store out_forest / out_numeric_counter MONITORING SERVER 具体的なConfiguration out_(growthforecast|munin |zabbix) LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward out_copy access log access log out_file_alternative access log error log error log error log out_parser nginx out_copy out_exec_filterWEB SERVERS out_(mail|mongo|redis out_exec_filter |file|mysql|s3) out_rewrite Mail / Store out_forest / out_numeric_counter out_(growthforecast|munin MONITORING |zabbix) SERVER LOG SERVER
    • fluent-agent-litefluent-plugin file-alternative numeric_counter parser growthforecast rewriteRecipe
    • fluent-agent-lite 難易度fluent-plugin 5もりす file-alternative numeric_counter 1あんちぽ parser rewrite growthforecastRecipe
    • fluent-agent-litefluent-plugin file-alternative numeric_counter parser growthforecast rewrite forestRecipe
    • fluent-agent-lite 難易度fluent-plugin 6もりす file-alternative numeric_counter 1あんちぽ parser rewrite growthforecast forestRecipe
    • fluent-agent-lite 難易度fluent-plugin lty h!!1 6モリス icu hinumeric_counter iff o g file-alternative D to is 1あんちぽ parser rewrite growthforecast forestRecipe
    • fluent-agent-lite 難易度 でも、fluent-plugin 少tyず h!!1 6モリス やれ iffic file-alternative D ばoo ul し inumeric_counter hg つ i s t大丈夫 1あんちぽ parser rewrite growthforecast !!1 forestRecipe
    • プロローグfluent-plugin 太古Recipe
    • fluent-pluginRecipe
    • access log access log cron rsync / scp access log error log error log error log nginx Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • access log access log e!! cron . ly.. ual rsync / scp ub l sio nm an Tro access log error log mis error log ans error log retr ed Ne nginx Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • access log access log e!! cron . ly.. ual rsync / scp ub l sio nm an Tro access log error log mis error log ans error log retr ed Ne nginx ha in!! C Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • fluent-agent-lite 第一章fluent-plugin file-alternative 生Recipe
    • fluent-agent-litefluent-plugin file-alternativeRecipe
    • access log access log cron rsync / scp access log error log error log error log nginx Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward access log access log out_file_alternative access log error log error log error log nginx Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • ・リアルタイムで転送たとえばサーバが突然死んでも、ログサーバで直前のログを参照できるプロセス自体が死ぬと・・・・途中でこけてもバッファリングによって再送プロセス自体が死ぬと・・・ scpやrsyncでまとめて転送するのと比べて何が優れているの か
    • ・リアルタイムで転送たとえばサーバが突然死んでも、ログサーバで直前のログを参照できるプロセス自体が死ぬと・・・・途中でこけてもバッファリングによって再送プロセス自体が死ぬと・・・ 視 は 監 う 対 しよ 絶 scpやrsyncでまとめて転送するのと比べて何が優れているの か
    • ・リアルタイムで転送たとえばサーバが突然死んでも、ログサーバで直前のログを参照できるプロセス自体が死ぬと・・・・途中でこけてもバッファリングによって再送 外道 視は 父プロセス自体が死ぬと・・・ 監 の匠 Flue う しよ ntd M http eetu ://blo p #2 対 mee g.fat 発表 tup- 資料 絶 vol0 her.g 2/ edow .net/ 2012 / 08/2 3/flue ntd- scpやrsyncでまとめて転送するのと比べて何が優れているの か
    • ・入れて起動したらほぼそのままでいい・パースしない ・正規表現を通さないので効率的 ・fluent-catは入力にJSONを要求するが・・・      「標準入力から読んでFluentdに送信するにはfluent-agent-liteが便利」       http://d.hatena.ne.jp/sfujiwara/20120611/1339373928・対象1ファイル毎に1プロセス起動する ・これは逆に注意点でもある Why fluent-agent-lite?
    • fluent-agent-lite 第二章fluent-plugin 老害 file-alternative parser forestRecipe
    • fluent-agent-litefluent-plugin file-alternative parser forestRecipe
    • td-agentfluent-agent-lite fluent-agent-lite in_forward access log access log out_file_alternative access log error log error log error log nginx Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward access log access log out_file_alternative access log error log error log error log nginx cron Midnight batch Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward access log access log out_file_alternative access log error log error log error log nginx cron QL BadS Midnight batch Serialize /WEB SERVERS Summarize / Analyze Mail / Store LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward access log access log out_file_alternative access log error log error log error log nginx cron QL BadS Midnight batch Serialize /WEB SERVERS Summarize / Too BIG LoaAnalyze dav !! g Mail / Store LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward out_copy access log access log out_file_alternative access log error log error log error log out_parser nginx out_exec_filterWEB SERVERS out_(mail|mongo|redis |file|mysql|s3) Mail / Store LOG SERVER
    • About out_exec_filter
    • td-agentfluent-agent-lite fluent-agent-lite in_forward out_copy access log access log out_file_alternative access log error log error log error log out_parser nginx out_exec_filterWEB SERVERS out_file MySQL mysqld_multi NEO replicate-do-table=a,e access log access log a b c a d e e DB SERVERS LOG SERVER
    • #!/usr/bin/env perl td-agent $sub = $log_column->{http_x_up_subno};use strict; }use warnings; elsif ( $log_column->{x-jphone-uid} ne - ) { fluent-agent-liteuse Data::Dumper; fluent-agent-lite in_forward$log_column->{x-jphone-uid}; $sub = out_copyuse DBIx::Handler; $sub =~ s/^.//; access loguse JSON::XS; } access loguse Data::MessagePack; else { print $mp->pack($log_column);$| = 1; out_file_alternative next;my $mp = Data::MessagePack->new(); } access log error log error logmy $db_user = root; error log my $sql = SELECT uid, update_time FROM trn_user WHERE sub = ?my $db_passwd = ; ORDER BY update_time DESC LIMIT 1;my $db_opt = { RaiseError => 0, PrintError => 1 }; out_parsermy $dsn1 = DBI:mysql:database=USERS1:localhost; my $even = $handler1->dbh->selectrow_hashref( $sql, undef, $sub );mysql_socket=/var/lib/mysql/mysql.sock6001; my $odd = $handler2->dbh->selectrow_hashref( $sql, undef, $sub );my $dsn2 = DBI:mysql:database=USERS2:localhost;mysql_socket=/var/lib/mysql/mysql.sock6002; my $UID = 0;$db_opt ); nginxmy $handler1 = DBIx::Handler->new( $dsn1, $db_user, $db_passwd, if ( defined $even && defined $odd ) { out_exec_filter $UID = $even->{update_time} ge $odd->{update_time} ? $even-my $handler2 = DBIx::Handler->new( $dsn2, $db_user, $db_passwd, >{uid} : $odd->{uid};$db_opt ); } elsif ( defined $even ) { WEB SERVERSwhile ( my $json = <STDIN> ) { $UID = $even->{uid}; my $log_column = eval { decode_json($json); }; } out_file next if ($@); elsif ( defined $odd ) { $UID = $odd->{uid}; unless ( defined $log_column->{ x-dcmguid || http_x_up_subno || x- }jphone-uid } ) { else { print $mp->pack($log_column); print $mp->pack($log_column); next; MySQL mysqld_multi next; } } NEO replicate-do-table=a,e access log access log my $sub; $log_column->{message} = $log_column->{message} . UID . $UID; a b c if ( $log_column->{x-dcmguid} ne - ) { a print $mp->pack($log_column); $sub = $log_column->{x-dcmguid}; } } d e e elsif ( $log_column->{http_x_up_subno} ne - ) { DB SERVERS LOG SERVER
    • ・途中の処理を得意な言語で書ける小さいコードでいいけど、しっかり書かないとエラったり予期せず死ぬので、運用エンジニアがコード書く練習するのに最適・JSONとMessagePack(とTSV)ただしVer.0.10.20以降・実行コストが高い(らしい)・out_formatにJSONはイケてないYajl::Parserがバッファリングするため About out_exec_filter
    • #!/usr/bin/env perluse strict;use warnings;use Data::Dumper;use JSON::XS;use Data::MessagePack;$| = 1;my $mp = Data::MessagePack->new();while ( my $json = <STDIN> ) { my $decode = eval { decode_json($json); }; next if ($@); unless ( defined $decode->{key} ) { print $mp->pack($decode); next; } print $mp->pack($decode);}My template script for out_exec_filter
    • fluent-agent-lite 第三章fluent-plugin parser 革命 file-alternative numeric_counter growthforecast forestRecipe
    • fluent-agent-litefluent-plugin file-alternative numeric_counter parser growthforecast forestRecipe
    • fluent-agent-litefluent-plugin file-alternative numeric_counter parser zabbix forestRecipe
    • fluent-agent-litefluent-plugin file-alternative numeric_counter parser munin forestRecipe
    • td-agentfluent-agent-lite fluent-agent-lite in_forward out_copy access log access log out_file_alternative access log error log error log error log out_parser nginx out_copy out_exec_filterWEB SERVERS out_(mail|mongo|redis out_exec_filter |file|mysql|s3) out_numeric_counter Mail / Store out_(growthforecast|munin |zabbix) MONITORING SERVER LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward out_copy access log access log out_file_alternative access log error log error log error log out_parser nginx out_copy out_exec_filterWEB SERVERS out_(mail|mongo|redis out_exec_filter |file|mysql|s3) out_sampling_filter out_numeric_counter Mail / Store out_(growthforecast|munin |zabbix) MONITORING SERVER LOG SERVER
    • fluent-agent-lite 最終章fluent-plugin parser 開眼 file-alternative numeric_counter growthforecast rewrite forestRecipe
    • fluent-agent-litefluent-plugin file-alternative numeric_counter parser growthforecast rewrite forestRecipe
    • td-agentfluent-agent-lite fluent-agent-lite in_forward out_copy access log access log out_file_alternative access log error log error log error log out_parser nginx out_copy out_exec_filterWEB SERVERS out_(mail|mongo|redis out_exec_filter |file|mysql|s3) out_numeric_counter Mail / Store out_(growthforecast|munin |zabbix) MONITORING SERVER LOG SERVER
    • td-agentfluent-agent-lite fluent-agent-lite in_forward out_copy access log access log out_file_alternative access log error log error log error log out_parser nginx out_copy out_exec_filterWEB SERVERS out_(mail|mongo|redis out_exec_filter |file|mysql|s3) out_rewrite Mail / Store out_numeric_counter out_(growthforecast|munin MONITORING |zabbix) SERVER LOG SERVER
    • nginx.access:{"method":"GET","path":"/mypage/status","status":"200","response_time":"161576"}Before rewrite
    • nginx.access:{"method":"GET","path":"/mypage/status","status":"200","response_time":"161576"}Before rewrite
    • nginx.access.mypage:{"method":"GET","path":"/mypage/status","status":"200","response_time":"161576"}After rewrite
    • Follow all request path<rule> pattern!!1 key path pattern ^/([^/]+) fallback others append_to_tag true</rule>Point of configuration
    • Follow all request path<rule> pattern!!1 key path patternthe next step ^/([^/]+) If out_datacounter, fallbackaware that there is an upper limit Be others to the number of patterns!!1 append_to_tag true</rule> Point of configuration
    • MySQL 番外編 slow_log 1 #!/usr/bin/env perl$ 27 $ tail 2$ 28 unless ( defined $slowlog->{sql} ) {$SLOW QUERY 3 use strict;$ 29 print $mp->pack($slowlog);$ 4 use warnings;$ 30 next;$ 5 use Data::Dumper;$ 31 }$ EXPLAIN 6 use DBIx::Handler;$ 7 use JSON::XS;$ 32 $ 33 my $select_statement;$ 8 use Data::MessagePack;$ 34 if ( $slowlog->{sql} =~ /(select[^;]+)/i ) {$ 9$ 35 $select_statement = $1;$ 10 $| = 1;$ 36 }$ 11 $ 37 $ 12 my $mp = Data::MessagePack->new();$ 38 if ( $slowlog->{sql} =~ /^use ([^;]+)/i ) {$ 13 $ 39 $db = ` . $1 . `;$ 14 my $db_user = "root";$ 40 }$ 15 my $db_passwd = "";$ 41 $ 16 my $db_opt = { RaiseError => 0, PrintError => 1 };$ 42 if ( defined $db && defined $select_statement ) {$ 17 my $handler = DBIx::Handler->new( "DBI:mysql: 43 $handler->dbh->do("use $db");$database=information_schema:1 27.0.0.1", $db_user, 44 my $explains = eval { $handler->dbh-$db_passwd, $db_opt );$ >selectall_arrayref( "EXPLAIN $selec t_statement", +{ 18 $ Slice => {} } ); };$ 19 my $db;$ 45 if ( ref $explains eq ARRAY ) {$ 20 my $long_query_time = $handler->dbh- 46 $slowlog->{explain} = $explains;$>selectrow_array($ 47 $slowlog->{database} = $db;$ 21 "SELECT VARIABLE_VALUE FROM 48 }$GLOBAL_VARIABLES WHERE VARIABLE_NAME = 49 }$LONG_Q UERY_TIME");$ 50 $ 22 $handler->dbh->do("SELECT SLEEP 51 print $mp->pack($slowlog);$($long_query_time)");$ 52 }$ 23 $ 24 while ( my $json = <STDIN> ) {$ 25 my $slowlog = eval { decode_json($json); };$ 26 next if ($@);$ store MongoDB
    • MySQL slow_log 1 #!/usr/bin/env perl$ 27 $ tail 2$ 28 unless ( defined $slowlog->{sql} ) {$ 3 use strict;$ 29 print $mp->pack($slowlog);$ 4 use warnings;$ 30 next;$ 5 use Data::Dumper;$ 31 }$ EXPLAIN 6 use DBIx::Handler;$ 7 use JSON::XS;$ 32 $ 33 my $select_statement;$ 8 use Data::MessagePack;$ 34 if ( $slowlog->{sql} =~ /(select[^;]+)/i ) {$ 9$ 35 $select_statement = $1;$ 10 $| = 1;$ 36 }$ 11 $ 37 $ 12 my $mp = Data::MessagePack->new();$ 38 if ( $slowlog->{sql} =~ /^use ([^;]+)/i ) {$ 13 $ 39 $db = ` . $1 . `;$ 14 my $db_user = "root";$ 40 }$ 15 my $db_passwd = "";$ 41 $ 16 my $db_opt = { RaiseError => 0, PrintError => 1 };$ 42 if ( defined $db && defined $select_statement ) {$ 17 my $handler = DBIx::Handler->new( "DBI:mysql: 43 $handler->dbh->do("use $db");$database=information_schema:1 27.0.0.1", $db_user, 44 my $explains = eval { $handler->dbh-$db_passwd, $db_opt );$ >selectall_arrayref( "EXPLAIN $selec t_statement", +{ 18 $ Slice => {} } ); };$ 19 my $db;$ 45 if ( ref $explains eq ARRAY ) {$ 20 my $long_query_time = $handler->dbh- 46 $slowlog->{explain} = $explains;$>selectrow_array($ 47 $slowlog->{database} = $db;$ 21 "SELECT VARIABLE_VALUE FROM 48 }$GLOBAL_VARIABLES WHERE VARIABLE_NAME = 49 }$LONG_Q UERY_TIME");$ 50 $ 22 $handler->dbh->do("SELECT SLEEP 51 print $mp->pack($slowlog);$($long_query_time)");$ 52 }$ 23 $ 24 while ( my $json = <STDIN> ) {$ 25 my $slowlog = eval { decode_json($json); };$ 26 next if ($@);$ store MongoDB
    • 公開したよ!https://github.com/studio3104/out-exec-mysql-slowquery
    • エピローグスーパー宣伝タイムOpeness is our driver for excellence
    • Talk 4th Section at Fluentd Meetup #3Small Startup Fluentd ご清聴ありがとうございました @studio3104