SL
Uploaded bySaewoong Lee
207 views

Fluentd 20150918 no_demo_public

The document outlines the implementation and features of Fluentd, covering its architecture, installation, and testing for MySQL slow query logging and process list logging. It includes detailed instructions on setting up Fluentd and configuring it for specific logging tasks, along with example commands and code snippets. Additionally, it discusses the integration of various plugins to enhance functionality and reliability in log collection.

Embed presentation

Download to read offline
fluentd LEE SAE WOONG 2015.09.18
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 2
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 3
Intro 4 • 1,500 persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement MySS architecture
Intro 5 • 1,500 persons • 20,000,000,000 yen 3nd Development ??? (real time slow query)
Intro 6 • 1,500 persons • 20,000,000,000 yen
Intro 7 Before fluentd Duplicated codes for error logic Complex codes for retry logic
Intro 8 After fluentd Integrated log collector
Intro 9 • 1,500 persons • 20,000,000,000 yen Scribe Flume Fluentd logstash
Intro 10 구분 강조점 유연성 / 상호운용성 단순함 / 안정성 구현언어 J 루비(Jruby) C루비(Cruby) 입력방법 많음 적음 플러그인 많음 매우 많음 플러그인 설치 쉬움 (jar 배포) 쉬움 (gem 설치) 신뢰성 메시지 loss 가능성 있음 메시지 loss 없음
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 11
Intro 12 Internal Architecture Input Output
Intro 13 Mysql : slow query Input Output +--------------------------------------------+ | sql_text | +--------------------------------------------+ | SET timestamp=1441820542; select sleep(1); | | SET timestamp=1441820574; select sleep(1); | +--------------------------------------------+
Intro 14 Mysql : slow query + LINE Input Output Parser parser Filter hostname +------------------+ | hostname | +------------------+ | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | +-----------------------+ +-------------+ | sql_text | +-------------+ | select sleep(1) | | select sleep(1) | | select sleep(1) | | select sleep(1) | +-------------+
Intro 15 Mysql : process list Input Output
Intro 16 493 plugins 2015.09.10 http://www.fluentd.org/plugins
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 17
Install 18 1. Install > curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh 2. Start demon > service td-agent start Starting td-agent: [ OK ] > service td-agent status td-agent (pid 2567) is running > service td-agent restart > service td-agent stop > td-agent --version td-agent 0.12.12 3. Transfer Sample Log > curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test.test > tail -n 1 /var/log/td-agent/td-agent.log 2015-09-10 00:26:31 +0900 debug.test.test: {"json":"message"}
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 19
Test 20 MySQL : slow query <source> type mysql_slow_query path /mysql/logs/slow_query.log pos_file /mysql/logs/slow_query.pos tag raw.collect.mysql_slow </source> <match raw.collect.mysql_slow> type parser remove_prefix raw format /^SET timestamp=(?<time_stamp>[^;]*); (?<sql>[^;]*);$/ time_format %d/%b/%Y:%H:%M:%S %z key_name sql reserve_data yes </match> <match collect.mysql_slow> type hostname key_name hostname add_prefix filtered </match> <match filtered.collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql_text key_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql table t_mysql_slow flush_interval 5s </store> </match>
Test 21 MySQL : slow query * Tail Log * MySQL Data
Test 22 MySQL : slow query # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_slow; create table DBPart.t_mysql_slow ( seq bigint auto_increment , hostname varchar(30) , log_date datetime , db varchar(100) , user varchar(100) , host varchar(100) , host_ip varchar(20) , query_time decimal(20,10) , lock_time decimal(20,10) , rows_sent bigint , rows_examined bigint , sql_text varchar(10000) , primary key(seq) , key(log_date) );
Test 23 MySQL : slow query # Client 1. fluentd plugin install * fluent-plugin-mysqlslowquery / fluent-plugin-mysql / fluent-plugin-mysql-bulk https://github.com/yuku-t/fluent-plugin-mysqlslowquery https://github.com/tagomoris/fluent-plugin-mysql https://github.com/toyama0919/fluent-plugin-mysql-bulk > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysqlslowquery > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-bulk 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
Test 24 MySQL : slow query > vi /etc/td-agent/td-agent.conf ========================================================= ... <source> type mysql_slow_query path /mysql/logs/slow_query.log tag collect.mysql_slow </source> <match collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.100 port 3306 database DBPart username root password rex column_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql_text key_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql table t_mysql_slow flush_interval 5s </store> </match> =========================================================
Test 25 MySQL : slow query 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. press slow query mysql -u root -p use mysql; select sleep(1); select sleep(1); [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log 2015-09-10 12:06:31 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000855,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854391; select sleep(1);","db":"mysql"} 2015-09-10 12:06:33 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000862,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854393; select sleep(1);","db":"mysql"} 2015-09-10 12:24:16 +0900 [info]: bulk insert values size => 2 * SERVER mysql> select * from t_mysql_slow; +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | log_date | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | hostname | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | 2015-09-10 07:03:45 | root[root] | localhost | | 1.0008600000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820542; select sleep(1); | NULL | | 2015-09-10 07:04:15 | root[root] | localhost | | 1.0010030000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820574; select sleep(1); | NULL | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ 8 rows in set (0.00 sec)
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 26
Test 27 MySQL : show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duration_time key_names hostname,Id,User,Host,db,Command,State,Info,duration_time table t_mysql_process flush_interval 5s </store> </match>
Test 28 * Tail Log * MySQL Data MySQL : show processlist
Test 29 MySQL : show processlist # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_process; create table DBPart.t_mysql_process ( hostname varchar(100) , log_date datetime default current_timestamp , id bigint , user varchar(100) , host varchar(100) , db varchar(64) , command varchar(50) , duration_time bigint , state varchar(4000) , info varchar(10000) );
Test 30 MySQL : show processlist # Client 1. fluentd plugin install * fluent-plugin-mysql-query / fluent-plugin-rename-key https://github.com/y-ken/fluent-plugin-mysql-query https://github.com/shunwen/fluent-plugin-rename-key > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-query > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-rename-key 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
Test 31 MySQL : show processlist > vi /etc/td-agent/td-agent.conf ========================================= ... ### mysql show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data #row_count yes #row_count_key row_count </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time key_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time table t_mysql_process flush_interval 5s </store> </match> =========================================
Test 32 MySQL : show processlist 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. show processlist; * client mysql -u root -p root@localhost:(none) 14:43:47>show processlist; +----+------+----------------------+--------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+----------------------+--------+---------+------+-------+------------------+ | 3 | root | localhost | NULL | Sleep | 381 | | NULL | | 7 | root | localhost | NULL | Query | 0 | init | show processlist | | 9 | root | 192.168.95.222:59359 | DBPart | Sleep | 43 | | NULL | | 10 | root | localhost | NULL | Sleep | 3 | | NULL | +----+------+----------------------+--------+---------+------+-------+------------------+ 4 rows in set (0.00 sec) [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log … 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"3","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"578","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"7","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"60","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"9","User":"root","Host":"192.168.95.222:59359","db":"DBPart","Command":"Query","duration_time":"0","State":"init","Info":"show full processlist"} 2015-09-10 14:47:30 +0900 [info]: bulk insert values size => 3 --> Send every 1 minutes
Test 33 MySQL : show processlist * SERVER root@localhost:DBPart 15:07:51>select * from t_mysql_process; +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | log_date | hostname | id | user | host | db | command | duration_time | state | info | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | 2015-09-10 15:04:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 338 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 25 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 398 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 17 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 20 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 458 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 16 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:06:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 80 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 518 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 7 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ 14 rows in set (0.00 sec)
Test 34 DEMO?
Test 35
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Error / Load test V. To do VI. Reference INDEX Agenda 36
Test 37 If client server (mysql) was down?
Test 38 Error Test Case. MySQL server down Error * client use mysql; select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server stop select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server start * server > select * from t_mysql_slow; +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | seq | hostname | log_date | db | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | 1 | NULL | NULL | mysql | root[root] | localhost | | 1.0015090000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862020; select sleep(1); | | 2 | NULL | NULL | mysql | root[root] | localhost | | 1.0004050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862021; select sleep(1); | | 3 | NULL | NULL | mysql | root[root] | localhost | | 1.0004840000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862022; select sleep(1); | | 4 | NULL | NULL | mysql | root[root] | localhost | | 1.0004360000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862023; select sleep(1); | | 5 | NULL | NULL | mysql | root[root] | localhost | | 1.0011230000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862024; select sleep(1); | | 6 | NULL | NULL | mysql | root[root] | localhost | | 1.0007080000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862025; select sleep(1); | | 7 | NULL | NULL | mysql | root[root] | localhost | | 1.0012720000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862026; select sleep(1); | | 8 | NULL | NULL | mysql | root[root] | localhost | | 1.0004110000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862027; select sleep(1); | | 9 | NULL | NULL | mysql | root[root] | localhost | | 1.0003540000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862028; select sleep(1); | | 10 | NULL | NULL | mysql | root[root] | localhost | | 1.0003870000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862029; select sleep(1); | | 11 | NULL | NULL | mysql | root[root] | localhost | | 1.0005050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862030; select sleep(1); | | 12 | NULL | NULL | mysql | root[root] | localhost | | 1.0005450000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862031; select sleep(1); | | 13 | NULL | NULL | mysql | root[root] | localhost | | 1.0002890000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862032; select sleep(1); | | 14 | NULL | NULL | mysql | root[root] | localhost | | 1.0011930000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862033; select sleep(1); | | 15 | NULL | NULL | mysql | root[root] | localhost | | 1.0010970000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862034; select sleep(1); | | 16 | NULL | NULL | mysql | root[root] | localhost | | 1.0012100000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862035; select sleep(1); | | 17 | NULL | NULL | mysql | root[root] | localhost | | 1.0012670000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862036; select sleep(1); | | 18 | NULL | NULL | mysql | root[root] | localhost | | 1.0007300000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862037; select sleep(1); | | 19 | NULL | NULL | mysql | root[root] | localhost | | 1.0007950000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862038; select sleep(1); | | 20 | NULL | NULL | mysql | root[root] | localhost | | 1.0010920000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862039; select sleep(1); | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ 20 rows in set (0.00 sec)
Test 39 Is Agent heavy?
Test 40 Load Test # press bash shell script > vi slow_log_sleep.sh ------------------------------------------------------------------------------------ #!/bin/bash i=0 x=120 while [ $i -ne $x ] do /mysql/bin/mysql -u root -p'rex' -e"select sleep(1);" i=$(( $i + 1 )) done ------------------------------------------------------------------------------------
Test 41 Load Test Case1. press slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 34k 99k| 0 0 | 0 0 0.3|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 1082B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 300B 592B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 332B 1086B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 332B 1032B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M
Test 42 Load Test Case1. press slow query * Pressing status - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 99 0 0 0| 32k 0 | 120B 782B|1.0 0 2.0|1114M 934M| 741M 7392k 92.7M 148M 0 1 99 0 0 0| 32k 28k| 120B 1438B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 1 97 0 1 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 0 99 0 0 0| 96k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 0 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 1 98 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 1 1 97 0 0 1| 64k 0 | 120B 782B|4.0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 1 96 3 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 64k 12k| 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 96 2 0 0| 64k 0 | 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 0 100 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 98 0 0 0| 32k 0 | 210B 836B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 32k 16k| 210B 836B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 798B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 1 1 93 5 0 0| 64k 0 | 180B 824B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 1 94 5 0 0| 96k 0 | 120B 782B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M
Test 43 Load Test Case2. press fluentd & slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 1242B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 1 98 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 212B 484B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M
Test 44 Load Test Case2. press fluentd & slow query * service td-agent start - cpu : idle 99~100% - mem : used 804M (60M used) - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 32k 95k| 0 0 | 0 0 0.3|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 1082B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 12 24 4 56 2 1|2692k 0 | 226B 670B| 0 1.0 24|1112M 936M| 766M 8004k 95.5M 120M <-- agent start 19 23 0 57 0 1|3444k 0 | 60B 628B|1.0 0 0|1112M 936M| 776M 8856k 98.0M 106M 7 54 0 36 1 1| 872k 6532k| 180B 2072B|1.0 0 9.0|1118M 930M| 827M 6988k 90.6M 64.6M 11 14 51 24 0 0|2448k 48k|1921B 5230B| 0 0 9.0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 458B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 240B 550B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M
Test 45 Load Test Case2. press fluentd & slow query * Pressing status - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 98 1 0 0| 32k 24k| 180B 1112B|1.0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.8M 0 1 97 2 0 0| 64k 24k| 240B 1810B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 97 1 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.7M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.4M 0 0 100 0 0 0| 64k 0 | 180B 1218B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 96 2 0 0| 32k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 0 0 100 0 0 0| 32k 0 | 180B 1128B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 1 1 98 0 0 0| 64k 0 | 818B 2587B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 1 99 0 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.2M 1 1 98 0 0 0| 64k 0 | 180B 1112B|1.0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 0 99 1 0 0| 64k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 1 96 2 0 0| 64k 4096B| 180B 1128B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 0 1 93 6 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 0 98 1 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 0 1 96 3 0 0| 64k 32k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7384k 84.9M 88.8M 1 1 96 2 0 0| 64k 8192B| 360B 1236B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.8M 0 1 95 2 1 1| 64k 0 | 818B 2603B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 1 0 99 0 0 0| 0 0 | 210B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 0 1 99 0 0 0| 32k 0 | 270B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M
Test 46 Load Test ### Conclusion * Case1 : no fluentd agent press - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K * Case2 : Fluentd agent press - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K >>> the load is not at all due to the agent. (only 60M) * 800,000 p/s logging
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 47
To do 48 • MySQL : slow query log • MySQL : process list • MySQL : show global status • Apache : web log • Linux : Syslog • Linux : df command • Linux : dstat command . . . . .
To do 49 • 1,500 persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement 3nd Development ???
To do 50
To do 51 master ETL slave master slave master slave master slave
52 MySQL : ETL <source> type mysql_query host rex port 3306 database test username root password rex interval 10s tag rex.log_game_play query select * from test.log_game_play where log_dat e between date_sub(now(), interval 10 minute) and now (); record_hostname yes nest_result no nest_key data </source> <match rex.log_game_play> type copy <store> type stdout </store> <store> type mysql_bulk host rex port 3306 database test username root password rex column_names log_date,useridx,play_time,char_type,result key_names log_date,useridx,play_time,char_type,result table log_game_play on_duplicate_key_update true on_duplicate_update_keys log_date,useridx flush_interval 15s </store> </match> To do
To do 53 MySS 6S or MySS 7
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 54
Reference 55 http://www.fluentd.org/casestudy/line
Reference 56 # RubyConf 2014 Build the Unified Logging Layer with Fluentd and Ruby by Kiyoto Tamura https://www.youtube.com/watch?v=sIVGsQgMHIo # Fluentd로 apache log를 MongoDB에 저장하기 http://www.sjune.net/archives/1164?ckattempt=1 # Big Data入門に見せかけたFluentd入門 http://www.slideshare.net/keithseahus/big-datafluentd # fluentd Plugins List http://www.fluentd.org/plugins/all http://www.fluentd.org/plugins https://github.com/search?utf8=✓&q=fluent-plugin+parser&type=Repositories&ref=searchresults # Fluentd - Pluggable log collector http://blog.seulgi.kim/2014/04/fluentd-pluggable-log-collector.html # Fluentd를 이용하여 모니터링도구 만들기 http://www.smallake.kr/?p=19241 # kibana https://www.elastic.co/products/kibana
thank you

More Related Content

PDF
Replacing Squid with ATS
byKit Chan
 
PDF
glance replicator
byirix_jp
 
PPTX
ProxySQL & PXC(Query routing and Failover Test)
byYoungHeon (Roy) Kim
 
PDF
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
byWei Lin
 
PPTX
MySQL Audit using Percona audit plugin and ELK
byYoungHeon (Roy) Kim
 
PPTX
Query logging with proxysql
byYoungHeon (Roy) Kim
 
PPTX
My sql failover test using orchestrator
byYoungHeon (Roy) Kim
 
PDF
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
byAltinity Ltd
 
Replacing Squid with ATS
byKit Chan
 
glance replicator
byirix_jp
 
ProxySQL & PXC(Query routing and Failover Test)
byYoungHeon (Roy) Kim
 
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
byWei Lin
 
MySQL Audit using Percona audit plugin and ELK
byYoungHeon (Roy) Kim
 
Query logging with proxysql
byYoungHeon (Roy) Kim
 
My sql failover test using orchestrator
byYoungHeon (Roy) Kim
 
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
byAltinity Ltd
 

What's hot

PDF
DATABASE AUTOMATION with Thousands of database, monitoring and backup
bySaewoong Lee
 
PDF
VCL template abstraction model and automated deployments to Fastly
byFastly
 
PDF
Script it
byGiuseppe Maxia
 
PDF
Using ngx_lua in UPYUN
byCong Zhang
 
PDF
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
byPuppet
 
PDF
Replication skeptic
byGiuseppe Maxia
 
PPTX
[오픈소스컨설팅] Linux Network Troubleshooting
byOpen Source Consulting
 
PPTX
Installation Openstack Swift
byymtech
 
PDF
Functional Operations (Functional Programming at Comcast Labs Connect)
bySusan Potter
 
PPT
{{more}} Kibana4
by琛琳 饶
 
PDF
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
byDocker, Inc.
 
PDF
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
byAltinity Ltd
 
PDF
SQL Track: Restoring databases with powershell
byITProceed
 
PDF
Dynamo: Not Just For Datastores
bySusan Potter
 
PPT
ELK stack at weibo.com
by琛琳 饶
 
PDF
Roll Your Own API Management Platform with nginx and Lua
byJon Moore
 
PDF
Lua tech talk
byLocaweb
 
PDF
Static Typing in Vault
byGlynnForrest
 
ODP
Puppet and Apache CloudStack
byPuppet
 
PDF
Moving Graphs to Production At Scale
byNeo4j
 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
bySaewoong Lee
 
VCL template abstraction model and automated deployments to Fastly
byFastly
 
Script it
byGiuseppe Maxia
 
Using ngx_lua in UPYUN
byCong Zhang
 
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
byPuppet
 
Replication skeptic
byGiuseppe Maxia
 
[오픈소스컨설팅] Linux Network Troubleshooting
byOpen Source Consulting
 
Installation Openstack Swift
byymtech
 
Functional Operations (Functional Programming at Comcast Labs Connect)
bySusan Potter
 
{{more}} Kibana4
by琛琳 饶
 
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
byDocker, Inc.
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
byAltinity Ltd
 
SQL Track: Restoring databases with powershell
byITProceed
 
Dynamo: Not Just For Datastores
bySusan Potter
 
ELK stack at weibo.com
by琛琳 饶
 
Roll Your Own API Management Platform with nginx and Lua
byJon Moore
 
Lua tech talk
byLocaweb
 
Static Typing in Vault
byGlynnForrest
 
Puppet and Apache CloudStack
byPuppet
 
Moving Graphs to Production At Scale
byNeo4j
 

Similar to Fluentd 20150918 no_demo_public

PDF
MySQL Tuning using digested slow-logs
byBob Burgess
 
PDF
Mysql tracing
byAnis Berejeb
 
PDF
Mysql tracing
byAnis Berejeb
 
DOCX
Performence tuning
byVasudeva Rao
 
PPT
PHP tips by a MYSQL DBA
byAmit Kumar Singh
 
PDF
Capturing, Analyzing and Optimizing MySQL
byRonald Bradford
 
PDF
Capturing, Analyzing, and Optimizing your SQL
byPadraig O'Sullivan
 
PDF
Maximizing SQL Reviews and Tuning with pt-query-digest
byPythian
 
PDF
Troubleshooting MySQL Performance
bySveta Smirnova
 
ODP
Beyond PHP - It's not (just) about the code
byWim Godden
 
PDF
16 MySQL Optimization #burningkeyboards
byDenis Ristic
 
PDF
Ten Reasons Why You Should Prefer PostgreSQL to MySQL
byanandology
 
PDF
Troubleshooting MySQL Performance
bySveta Smirnova
 
PPTX
MySQL performance tuning
byAnurag Srivastava
 
ODP
Beyond PHP - it's not (just) about the code
byWim Godden
 
PDF
Tracing and profiling my sql (percona live europe 2019) draft_1
byValerii Kravchuk
 
PDF
MySQL Performance Schema in 20 Minutes
bySveta Smirnova
 
PDF
MySQL performance webinar
byAbel Flórez
 
PDF
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
byJaime Crespo
 
PDF
Secondary Index Search in InnoDB
byMIJIN AN
 
MySQL Tuning using digested slow-logs
byBob Burgess
 
Mysql tracing
byAnis Berejeb
 
Mysql tracing
byAnis Berejeb
 
Performence tuning
byVasudeva Rao
 
PHP tips by a MYSQL DBA
byAmit Kumar Singh
 
Capturing, Analyzing and Optimizing MySQL
byRonald Bradford
 
Capturing, Analyzing, and Optimizing your SQL
byPadraig O'Sullivan
 
Maximizing SQL Reviews and Tuning with pt-query-digest
byPythian
 
Troubleshooting MySQL Performance
bySveta Smirnova
 
Beyond PHP - It's not (just) about the code
byWim Godden
 
16 MySQL Optimization #burningkeyboards
byDenis Ristic
 
Ten Reasons Why You Should Prefer PostgreSQL to MySQL
byanandology
 
Troubleshooting MySQL Performance
bySveta Smirnova
 
MySQL performance tuning
byAnurag Srivastava
 
Beyond PHP - it's not (just) about the code
byWim Godden
 
Tracing and profiling my sql (percona live europe 2019) draft_1
byValerii Kravchuk
 
MySQL Performance Schema in 20 Minutes
bySveta Smirnova
 
MySQL performance webinar
byAbel Flórez
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
byJaime Crespo
 
Secondary Index Search in InnoDB
byMIJIN AN
 

Recently uploaded

PPTX
Unit 1 Introduction to Information Technology in Business.pptx
bygoitranjan001
 
PDF
Narrows Planning Collective Transportation Capstone.pdf
bycj2392
 
PDF
Lecture -06-Hybrid Policies - Chapter 7- Weeks 6-7.pdf
bysalaarmike
 
PDF
Presentation-on-Energy-Transition-in-Bangladesh-Employment-and-Skills.pdf
bysyedSajib8
 
PPTX
Design and Thermal Analysis of Plate Type Heat Exchanger
byKhageshKumarDansena
 
PPTX
Coupled Hazard-Mobility Framework for Optimizing Emergency Vehicle Routing
byghn2015
 
PPTX
firewall Selection in production life pptx
byssuserb1479b
 
PDF
CV_Rahul_ELV Project Engineer_8 Years Experience
byRahulSushruth
 
PDF
STOP CONSOLIDATION: Optimizing Transit Speed & Rider Experience
byti2208
 
PPTX
Basic Electrical Engineering-Introduction to Light Sources
byKhuman Haobam
 
PPTX
How to Use Mobile CMMS to Improve Maintenance Operations & Field Productivity
byMaintWiz Technologies Private Limited
 
PPTX
Emerging Trends and Research Frontiers in Chemical Engineering for Green and ...
byChemical Engineering Dept. NIT Rourkela-769008, Odisha, India
 
PPTX
Importance of Asset Register in Maintenance Management
byMaintWiz Technologies Private Limited
 
PPTX
ISO 14224 Compliance & CMMS Software — A Comprehensive Guide for Reliable Mai...
byMaintWiz Technologies Private Limited
 
PPTX
Best CMMS for IoT Integration: Real-Time Asset Intelligence & Smart Maintenan...
byMaintWiz Technologies Private Limited
 
PPTX
DEEP LEARNING UNIT 2 all about deep learning
bypavanforsocials
 
PPTX
Natural Gas fundamentals and GRU for associated gas trap.pptx
byengineerhassan
 
PPTX
Salesforce Bulk Connector V1 and V2 Deep Dive!
byRajeevRanjan368982
 
PPTX
Plant Performance Strategies: Enhanced Reliability & Operational Efficiency w...
byMaintWiz Technologies Private Limited
 
PDF
ACI 318-2205_American Concrete Institute.pdf
byericaguilerasoto1
 
Unit 1 Introduction to Information Technology in Business.pptx
bygoitranjan001
 
Narrows Planning Collective Transportation Capstone.pdf
bycj2392
 
Lecture -06-Hybrid Policies - Chapter 7- Weeks 6-7.pdf
bysalaarmike
 
Presentation-on-Energy-Transition-in-Bangladesh-Employment-and-Skills.pdf
bysyedSajib8
 
Design and Thermal Analysis of Plate Type Heat Exchanger
byKhageshKumarDansena
 
Coupled Hazard-Mobility Framework for Optimizing Emergency Vehicle Routing
byghn2015
 
firewall Selection in production life pptx
byssuserb1479b
 
CV_Rahul_ELV Project Engineer_8 Years Experience
byRahulSushruth
 
STOP CONSOLIDATION: Optimizing Transit Speed & Rider Experience
byti2208
 
Basic Electrical Engineering-Introduction to Light Sources
byKhuman Haobam
 
How to Use Mobile CMMS to Improve Maintenance Operations & Field Productivity
byMaintWiz Technologies Private Limited
 
Emerging Trends and Research Frontiers in Chemical Engineering for Green and ...
byChemical Engineering Dept. NIT Rourkela-769008, Odisha, India
 
Importance of Asset Register in Maintenance Management
byMaintWiz Technologies Private Limited
 
ISO 14224 Compliance & CMMS Software — A Comprehensive Guide for Reliable Mai...
byMaintWiz Technologies Private Limited
 
Best CMMS for IoT Integration: Real-Time Asset Intelligence & Smart Maintenan...
byMaintWiz Technologies Private Limited
 
DEEP LEARNING UNIT 2 all about deep learning
bypavanforsocials
 
Natural Gas fundamentals and GRU for associated gas trap.pptx
byengineerhassan
 
Salesforce Bulk Connector V1 and V2 Deep Dive!
byRajeevRanjan368982
 
Plant Performance Strategies: Enhanced Reliability & Operational Efficiency w...
byMaintWiz Technologies Private Limited
 
ACI 318-2205_American Concrete Institute.pdf
byericaguilerasoto1
 

Fluentd 20150918 no_demo_public

  • 1.
  • 2.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 2
  • 3.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 3
  • 4.
    Intro 4 • 1,500persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement MySS architecture
  • 5.
    Intro 5 • 1,500persons • 20,000,000,000 yen 3nd Development ??? (real time slow query)
  • 6.
    Intro 6 • 1,500persons • 20,000,000,000 yen
  • 7.
    Intro 7 Before fluentd Duplicatedcodes for error logic Complex codes for retry logic
  • 8.
  • 9.
    Intro 9 • 1,500persons • 20,000,000,000 yen Scribe Flume Fluentd logstash
  • 10.
    Intro 10 구분 강조점 유연성/ 상호운용성 단순함 / 안정성 구현언어 J 루비(Jruby) C루비(Cruby) 입력방법 많음 적음 플러그인 많음 매우 많음 플러그인 설치 쉬움 (jar 배포) 쉬움 (gem 설치) 신뢰성 메시지 loss 가능성 있음 메시지 loss 없음
  • 11.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 11
  • 12.
  • 13.
    Intro 13 Mysql :slow query Input Output +--------------------------------------------+ | sql_text | +--------------------------------------------+ | SET timestamp=1441820542; select sleep(1); | | SET timestamp=1441820574; select sleep(1); | +--------------------------------------------+
  • 14.
    Intro 14 Mysql :slow query + LINE Input Output Parser parser Filter hostname +------------------+ | hostname | +------------------+ | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | +-----------------------+ +-------------+ | sql_text | +-------------+ | select sleep(1) | | select sleep(1) | | select sleep(1) | | select sleep(1) | +-------------+
  • 15.
    Intro 15 Mysql :process list Input Output
  • 16.
  • 17.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 17
  • 18.
    Install 18 1. Install >curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh 2. Start demon > service td-agent start Starting td-agent: [ OK ] > service td-agent status td-agent (pid 2567) is running > service td-agent restart > service td-agent stop > td-agent --version td-agent 0.12.12 3. Transfer Sample Log > curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test.test > tail -n 1 /var/log/td-agent/td-agent.log 2015-09-10 00:26:31 +0900 debug.test.test: {"json":"message"}
  • 19.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 19
  • 20.
    Test 20 MySQL :slow query <source> type mysql_slow_query path /mysql/logs/slow_query.log pos_file /mysql/logs/slow_query.pos tag raw.collect.mysql_slow </source> <match raw.collect.mysql_slow> type parser remove_prefix raw format /^SET timestamp=(?<time_stamp>[^;]*); (?<sql>[^;]*);$/ time_format %d/%b/%Y:%H:%M:%S %z key_name sql reserve_data yes </match> <match collect.mysql_slow> type hostname key_name hostname add_prefix filtered </match> <match filtered.collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql_text key_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql table t_mysql_slow flush_interval 5s </store> </match>
  • 21.
    Test 21 MySQL :slow query * Tail Log * MySQL Data
  • 22.
    Test 22 MySQL :slow query # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_slow; create table DBPart.t_mysql_slow ( seq bigint auto_increment , hostname varchar(30) , log_date datetime , db varchar(100) , user varchar(100) , host varchar(100) , host_ip varchar(20) , query_time decimal(20,10) , lock_time decimal(20,10) , rows_sent bigint , rows_examined bigint , sql_text varchar(10000) , primary key(seq) , key(log_date) );
  • 23.
    Test 23 MySQL :slow query # Client 1. fluentd plugin install * fluent-plugin-mysqlslowquery / fluent-plugin-mysql / fluent-plugin-mysql-bulk https://github.com/yuku-t/fluent-plugin-mysqlslowquery https://github.com/tagomoris/fluent-plugin-mysql https://github.com/toyama0919/fluent-plugin-mysql-bulk > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysqlslowquery > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-bulk 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
  • 24.
    Test 24 MySQL :slow query > vi /etc/td-agent/td-agent.conf ========================================================= ... <source> type mysql_slow_query path /mysql/logs/slow_query.log tag collect.mysql_slow </source> <match collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.100 port 3306 database DBPart username root password rex column_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql_text key_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql table t_mysql_slow flush_interval 5s </store> </match> =========================================================
  • 25.
    Test 25 MySQL :slow query 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. press slow query mysql -u root -p use mysql; select sleep(1); select sleep(1); [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log 2015-09-10 12:06:31 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000855,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854391; select sleep(1);","db":"mysql"} 2015-09-10 12:06:33 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000862,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854393; select sleep(1);","db":"mysql"} 2015-09-10 12:24:16 +0900 [info]: bulk insert values size => 2 * SERVER mysql> select * from t_mysql_slow; +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | log_date | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | hostname | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | 2015-09-10 07:03:45 | root[root] | localhost | | 1.0008600000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820542; select sleep(1); | NULL | | 2015-09-10 07:04:15 | root[root] | localhost | | 1.0010030000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820574; select sleep(1); | NULL | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ 8 rows in set (0.00 sec)
  • 26.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 26
  • 27.
    Test 27 MySQL :show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duration_time key_names hostname,Id,User,Host,db,Command,State,Info,duration_time table t_mysql_process flush_interval 5s </store> </match>
  • 28.
    Test 28 * TailLog * MySQL Data MySQL : show processlist
  • 29.
    Test 29 MySQL :show processlist # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_process; create table DBPart.t_mysql_process ( hostname varchar(100) , log_date datetime default current_timestamp , id bigint , user varchar(100) , host varchar(100) , db varchar(64) , command varchar(50) , duration_time bigint , state varchar(4000) , info varchar(10000) );
  • 30.
    Test 30 MySQL :show processlist # Client 1. fluentd plugin install * fluent-plugin-mysql-query / fluent-plugin-rename-key https://github.com/y-ken/fluent-plugin-mysql-query https://github.com/shunwen/fluent-plugin-rename-key > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-query > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-rename-key 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
  • 31.
    Test 31 MySQL :show processlist > vi /etc/td-agent/td-agent.conf ========================================= ... ### mysql show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data #row_count yes #row_count_key row_count </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time key_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time table t_mysql_process flush_interval 5s </store> </match> =========================================
  • 32.
    Test 32 MySQL :show processlist 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. show processlist; * client mysql -u root -p root@localhost:(none) 14:43:47>show processlist; +----+------+----------------------+--------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+----------------------+--------+---------+------+-------+------------------+ | 3 | root | localhost | NULL | Sleep | 381 | | NULL | | 7 | root | localhost | NULL | Query | 0 | init | show processlist | | 9 | root | 192.168.95.222:59359 | DBPart | Sleep | 43 | | NULL | | 10 | root | localhost | NULL | Sleep | 3 | | NULL | +----+------+----------------------+--------+---------+------+-------+------------------+ 4 rows in set (0.00 sec) [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log … 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"3","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"578","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"7","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"60","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"9","User":"root","Host":"192.168.95.222:59359","db":"DBPart","Command":"Query","duration_time":"0","State":"init","Info":"show full processlist"} 2015-09-10 14:47:30 +0900 [info]: bulk insert values size => 3 --> Send every 1 minutes
  • 33.
    Test 33 MySQL :show processlist * SERVER root@localhost:DBPart 15:07:51>select * from t_mysql_process; +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | log_date | hostname | id | user | host | db | command | duration_time | state | info | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | 2015-09-10 15:04:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 338 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 25 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 398 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 17 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 20 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 458 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 16 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:06:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 80 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 518 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 7 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ 14 rows in set (0.00 sec)
  • 34.
  • 35.
  • 36.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Error / Load test V. To do VI. Reference INDEX Agenda 36
  • 37.
    Test 37 If clientserver (mysql) was down?
  • 38.
    Test 38 Error Test Case.MySQL server down Error * client use mysql; select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server stop select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server start * server > select * from t_mysql_slow; +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | seq | hostname | log_date | db | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | 1 | NULL | NULL | mysql | root[root] | localhost | | 1.0015090000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862020; select sleep(1); | | 2 | NULL | NULL | mysql | root[root] | localhost | | 1.0004050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862021; select sleep(1); | | 3 | NULL | NULL | mysql | root[root] | localhost | | 1.0004840000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862022; select sleep(1); | | 4 | NULL | NULL | mysql | root[root] | localhost | | 1.0004360000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862023; select sleep(1); | | 5 | NULL | NULL | mysql | root[root] | localhost | | 1.0011230000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862024; select sleep(1); | | 6 | NULL | NULL | mysql | root[root] | localhost | | 1.0007080000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862025; select sleep(1); | | 7 | NULL | NULL | mysql | root[root] | localhost | | 1.0012720000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862026; select sleep(1); | | 8 | NULL | NULL | mysql | root[root] | localhost | | 1.0004110000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862027; select sleep(1); | | 9 | NULL | NULL | mysql | root[root] | localhost | | 1.0003540000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862028; select sleep(1); | | 10 | NULL | NULL | mysql | root[root] | localhost | | 1.0003870000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862029; select sleep(1); | | 11 | NULL | NULL | mysql | root[root] | localhost | | 1.0005050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862030; select sleep(1); | | 12 | NULL | NULL | mysql | root[root] | localhost | | 1.0005450000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862031; select sleep(1); | | 13 | NULL | NULL | mysql | root[root] | localhost | | 1.0002890000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862032; select sleep(1); | | 14 | NULL | NULL | mysql | root[root] | localhost | | 1.0011930000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862033; select sleep(1); | | 15 | NULL | NULL | mysql | root[root] | localhost | | 1.0010970000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862034; select sleep(1); | | 16 | NULL | NULL | mysql | root[root] | localhost | | 1.0012100000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862035; select sleep(1); | | 17 | NULL | NULL | mysql | root[root] | localhost | | 1.0012670000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862036; select sleep(1); | | 18 | NULL | NULL | mysql | root[root] | localhost | | 1.0007300000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862037; select sleep(1); | | 19 | NULL | NULL | mysql | root[root] | localhost | | 1.0007950000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862038; select sleep(1); | | 20 | NULL | NULL | mysql | root[root] | localhost | | 1.0010920000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862039; select sleep(1); | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ 20 rows in set (0.00 sec)
  • 39.
  • 40.
    Test 40 Load Test #press bash shell script > vi slow_log_sleep.sh ------------------------------------------------------------------------------------ #!/bin/bash i=0 x=120 while [ $i -ne $x ] do /mysql/bin/mysql -u root -p'rex' -e"select sleep(1);" i=$(( $i + 1 )) done ------------------------------------------------------------------------------------
  • 41.
    Test 41 Load Test Case1.press slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 34k 99k| 0 0 | 0 0 0.3|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 1082B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 300B 592B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 332B 1086B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 332B 1032B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M
  • 42.
    Test 42 Load Test Case1.press slow query * Pressing status - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 99 0 0 0| 32k 0 | 120B 782B|1.0 0 2.0|1114M 934M| 741M 7392k 92.7M 148M 0 1 99 0 0 0| 32k 28k| 120B 1438B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 1 97 0 1 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 0 99 0 0 0| 96k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 0 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 1 98 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 1 1 97 0 0 1| 64k 0 | 120B 782B|4.0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 1 96 3 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 64k 12k| 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 96 2 0 0| 64k 0 | 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 0 100 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 98 0 0 0| 32k 0 | 210B 836B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 32k 16k| 210B 836B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 798B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 1 1 93 5 0 0| 64k 0 | 180B 824B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 1 94 5 0 0| 96k 0 | 120B 782B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M
  • 43.
    Test 43 Load Test Case2.press fluentd & slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 1242B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 1 98 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 212B 484B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M
  • 44.
    Test 44 Load Test Case2.press fluentd & slow query * service td-agent start - cpu : idle 99~100% - mem : used 804M (60M used) - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 32k 95k| 0 0 | 0 0 0.3|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 1082B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 12 24 4 56 2 1|2692k 0 | 226B 670B| 0 1.0 24|1112M 936M| 766M 8004k 95.5M 120M <-- agent start 19 23 0 57 0 1|3444k 0 | 60B 628B|1.0 0 0|1112M 936M| 776M 8856k 98.0M 106M 7 54 0 36 1 1| 872k 6532k| 180B 2072B|1.0 0 9.0|1118M 930M| 827M 6988k 90.6M 64.6M 11 14 51 24 0 0|2448k 48k|1921B 5230B| 0 0 9.0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 458B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 240B 550B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M
  • 45.
    Test 45 Load Test Case2.press fluentd & slow query * Pressing status - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 98 1 0 0| 32k 24k| 180B 1112B|1.0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.8M 0 1 97 2 0 0| 64k 24k| 240B 1810B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 97 1 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.7M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.4M 0 0 100 0 0 0| 64k 0 | 180B 1218B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 96 2 0 0| 32k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 0 0 100 0 0 0| 32k 0 | 180B 1128B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 1 1 98 0 0 0| 64k 0 | 818B 2587B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 1 99 0 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.2M 1 1 98 0 0 0| 64k 0 | 180B 1112B|1.0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 0 99 1 0 0| 64k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 1 96 2 0 0| 64k 4096B| 180B 1128B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 0 1 93 6 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 0 98 1 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 0 1 96 3 0 0| 64k 32k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7384k 84.9M 88.8M 1 1 96 2 0 0| 64k 8192B| 360B 1236B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.8M 0 1 95 2 1 1| 64k 0 | 818B 2603B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 1 0 99 0 0 0| 0 0 | 210B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 0 1 99 0 0 0| 32k 0 | 270B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M
  • 46.
    Test 46 Load Test ###Conclusion * Case1 : no fluentd agent press - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K * Case2 : Fluentd agent press - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K >>> the load is not at all due to the agent. (only 60M) * 800,000 p/s logging
  • 47.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 47
  • 48.
    To do 48 •MySQL : slow query log • MySQL : process list • MySQL : show global status • Apache : web log • Linux : Syslog • Linux : df command • Linux : dstat command . . . . .
  • 49.
    To do 49 •1,500 persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement 3nd Development ???
  • 50.
  • 51.
    To do 51 master ETL slave masterslave master slave master slave
  • 52.
    52 MySQL : ETL <source> typemysql_query host rex port 3306 database test username root password rex interval 10s tag rex.log_game_play query select * from test.log_game_play where log_dat e between date_sub(now(), interval 10 minute) and now (); record_hostname yes nest_result no nest_key data </source> <match rex.log_game_play> type copy <store> type stdout </store> <store> type mysql_bulk host rex port 3306 database test username root password rex column_names log_date,useridx,play_time,char_type,result key_names log_date,useridx,play_time,char_type,result table log_game_play on_duplicate_key_update true on_duplicate_update_keys log_date,useridx flush_interval 15s </store> </match> To do
  • 53.
    To do 53 MySS6S or MySS 7
  • 54.
    I. Intro II. Architecture III.Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 54
  • 55.
  • 56.
    Reference 56 # RubyConf2014 Build the Unified Logging Layer with Fluentd and Ruby by Kiyoto Tamura https://www.youtube.com/watch?v=sIVGsQgMHIo # Fluentd로 apache log를 MongoDB에 저장하기 http://www.sjune.net/archives/1164?ckattempt=1 # Big Data入門に見せかけたFluentd入門 http://www.slideshare.net/keithseahus/big-datafluentd # fluentd Plugins List http://www.fluentd.org/plugins/all http://www.fluentd.org/plugins https://github.com/search?utf8=✓&q=fluent-plugin+parser&type=Repositories&ref=searchresults # Fluentd - Pluggable log collector http://blog.seulgi.kim/2014/04/fluentd-pluggable-log-collector.html # Fluentd를 이용하여 모니터링도구 만들기 http://www.smallake.kr/?p=19241 # kibana https://www.elastic.co/products/kibana
  • 57.