SlideShare a Scribd company logo

Fluentd 20150918 no_demo_public

•
•
S
Saewoong Lee

I explain 'Fluentd' for mysql. - About fluentd - Test : MySQL slowquery logging - Test : MySQL processlist logging - Test : Load test

Engineering
1 of 57
Download to read offline
fluentd LEE SAE WOONG 2015.09.18
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 2
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 3
Intro 4 • 1,500 persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement MySS architecture
Intro 5 • 1,500 persons • 20,000,000,000 yen 3nd Development ??? (real time slow query)
Intro 6 • 1,500 persons • 20,000,000,000 yen
Intro 7 Before fluentd Duplicated codes for error logic Complex codes for retry logic
Intro 8 After fluentd Integrated log collector
Intro 9 • 1,500 persons • 20,000,000,000 yen Scribe Flume Fluentd logstash
Intro 10 구분 강조점 유연성 / 상호운용성 단순함 / 안정성 구현언어 J 루비(Jruby) C루비(Cruby) 입력방법 많음 적음 플러그인 많음 매우 많음 플러그인 설치 쉬움 (jar 배포) 쉬움 (gem 설치) 신뢰성 메시지 loss 가능성 있음 메시지 loss 없음
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 11
Intro 12 Internal Architecture Input Output
Intro 13 Mysql : slow query Input Output +--------------------------------------------+ | sql_text | +--------------------------------------------+ | SET timestamp=1441820542; select sleep(1); | | SET timestamp=1441820574; select sleep(1); | +--------------------------------------------+
Intro 14 Mysql : slow query + LINE Input Output Parser parser Filter hostname +------------------+ | hostname | +------------------+ | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | +-----------------------+ +-------------+ | sql_text | +-------------+ | select sleep(1) | | select sleep(1) | | select sleep(1) | | select sleep(1) | +-------------+
Intro 15 Mysql : process list Input Output
Intro 16 493 plugins 2015.09.10 http://www.fluentd.org/plugins
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 17
Install 18 1. Install > curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh 2. Start demon > service td-agent start Starting td-agent: [ OK ] > service td-agent status td-agent (pid 2567) is running > service td-agent restart > service td-agent stop > td-agent --version td-agent 0.12.12 3. Transfer Sample Log > curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test.test > tail -n 1 /var/log/td-agent/td-agent.log 2015-09-10 00:26:31 +0900 debug.test.test: {"json":"message"}
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 19
Test 20 MySQL : slow query <source> type mysql_slow_query path /mysql/logs/slow_query.log pos_file /mysql/logs/slow_query.pos tag raw.collect.mysql_slow </source> <match raw.collect.mysql_slow> type parser remove_prefix raw format /^SET timestamp=(?<time_stamp>[^;]*); (?<sql>[^;]*);$/ time_format %d/%b/%Y:%H:%M:%S %z key_name sql reserve_data yes </match> <match collect.mysql_slow> type hostname key_name hostname add_prefix filtered </match> <match filtered.collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql_text key_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql table t_mysql_slow flush_interval 5s </store> </match>
Test 21 MySQL : slow query * Tail Log * MySQL Data
Test 22 MySQL : slow query # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_slow; create table DBPart.t_mysql_slow ( seq bigint auto_increment , hostname varchar(30) , log_date datetime , db varchar(100) , user varchar(100) , host varchar(100) , host_ip varchar(20) , query_time decimal(20,10) , lock_time decimal(20,10) , rows_sent bigint , rows_examined bigint , sql_text varchar(10000) , primary key(seq) , key(log_date) );
Test 23 MySQL : slow query # Client 1. fluentd plugin install * fluent-plugin-mysqlslowquery / fluent-plugin-mysql / fluent-plugin-mysql-bulk https://github.com/yuku-t/fluent-plugin-mysqlslowquery https://github.com/tagomoris/fluent-plugin-mysql https://github.com/toyama0919/fluent-plugin-mysql-bulk > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysqlslowquery > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-bulk 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
Test 24 MySQL : slow query > vi /etc/td-agent/td-agent.conf ========================================================= ... <source> type mysql_slow_query path /mysql/logs/slow_query.log tag collect.mysql_slow </source> <match collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.100 port 3306 database DBPart username root password rex column_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql_text key_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql table t_mysql_slow flush_interval 5s </store> </match> =========================================================
Test 25 MySQL : slow query 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. press slow query mysql -u root -p use mysql; select sleep(1); select sleep(1); [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log 2015-09-10 12:06:31 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000855,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854391; select sleep(1);","db":"mysql"} 2015-09-10 12:06:33 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000862,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854393; select sleep(1);","db":"mysql"} 2015-09-10 12:24:16 +0900 [info]: bulk insert values size => 2 * SERVER mysql> select * from t_mysql_slow; +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | log_date | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | hostname | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | 2015-09-10 07:03:45 | root[root] | localhost | | 1.0008600000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820542; select sleep(1); | NULL | | 2015-09-10 07:04:15 | root[root] | localhost | | 1.0010030000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820574; select sleep(1); | NULL | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ 8 rows in set (0.00 sec)
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 26
Test 27 MySQL : show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duration_time key_names hostname,Id,User,Host,db,Command,State,Info,duration_time table t_mysql_process flush_interval 5s </store> </match>
Test 28 * Tail Log * MySQL Data MySQL : show processlist
Test 29 MySQL : show processlist # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_process; create table DBPart.t_mysql_process ( hostname varchar(100) , log_date datetime default current_timestamp , id bigint , user varchar(100) , host varchar(100) , db varchar(64) , command varchar(50) , duration_time bigint , state varchar(4000) , info varchar(10000) );
Test 30 MySQL : show processlist # Client 1. fluentd plugin install * fluent-plugin-mysql-query / fluent-plugin-rename-key https://github.com/y-ken/fluent-plugin-mysql-query https://github.com/shunwen/fluent-plugin-rename-key > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-query > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-rename-key 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
Test 31 MySQL : show processlist > vi /etc/td-agent/td-agent.conf ========================================= ... ### mysql show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data #row_count yes #row_count_key row_count </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time key_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time table t_mysql_process flush_interval 5s </store> </match> =========================================
Test 32 MySQL : show processlist 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. show processlist; * client mysql -u root -p root@localhost:(none) 14:43:47>show processlist; +----+------+----------------------+--------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+----------------------+--------+---------+------+-------+------------------+ | 3 | root | localhost | NULL | Sleep | 381 | | NULL | | 7 | root | localhost | NULL | Query | 0 | init | show processlist | | 9 | root | 192.168.95.222:59359 | DBPart | Sleep | 43 | | NULL | | 10 | root | localhost | NULL | Sleep | 3 | | NULL | +----+------+----------------------+--------+---------+------+-------+------------------+ 4 rows in set (0.00 sec) [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log … 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"3","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"578","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"7","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"60","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"9","User":"root","Host":"192.168.95.222:59359","db":"DBPart","Command":"Query","duration_time":"0","State":"init","Info":"show full processlist"} 2015-09-10 14:47:30 +0900 [info]: bulk insert values size => 3 --> Send every 1 minutes
Test 33 MySQL : show processlist * SERVER root@localhost:DBPart 15:07:51>select * from t_mysql_process; +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | log_date | hostname | id | user | host | db | command | duration_time | state | info | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | 2015-09-10 15:04:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 338 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 25 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 398 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 17 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 20 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 458 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 16 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:06:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 80 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 518 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 7 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ 14 rows in set (0.00 sec)
Test 34 DEMO?
Test 35
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Error / Load test V. To do VI. Reference INDEX Agenda 36
Test 37 If client server (mysql) was down?
Test 38 Error Test Case. MySQL server down Error * client use mysql; select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server stop select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server start * server > select * from t_mysql_slow; +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | seq | hostname | log_date | db | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | 1 | NULL | NULL | mysql | root[root] | localhost | | 1.0015090000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862020; select sleep(1); | | 2 | NULL | NULL | mysql | root[root] | localhost | | 1.0004050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862021; select sleep(1); | | 3 | NULL | NULL | mysql | root[root] | localhost | | 1.0004840000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862022; select sleep(1); | | 4 | NULL | NULL | mysql | root[root] | localhost | | 1.0004360000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862023; select sleep(1); | | 5 | NULL | NULL | mysql | root[root] | localhost | | 1.0011230000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862024; select sleep(1); | | 6 | NULL | NULL | mysql | root[root] | localhost | | 1.0007080000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862025; select sleep(1); | | 7 | NULL | NULL | mysql | root[root] | localhost | | 1.0012720000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862026; select sleep(1); | | 8 | NULL | NULL | mysql | root[root] | localhost | | 1.0004110000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862027; select sleep(1); | | 9 | NULL | NULL | mysql | root[root] | localhost | | 1.0003540000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862028; select sleep(1); | | 10 | NULL | NULL | mysql | root[root] | localhost | | 1.0003870000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862029; select sleep(1); | | 11 | NULL | NULL | mysql | root[root] | localhost | | 1.0005050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862030; select sleep(1); | | 12 | NULL | NULL | mysql | root[root] | localhost | | 1.0005450000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862031; select sleep(1); | | 13 | NULL | NULL | mysql | root[root] | localhost | | 1.0002890000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862032; select sleep(1); | | 14 | NULL | NULL | mysql | root[root] | localhost | | 1.0011930000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862033; select sleep(1); | | 15 | NULL | NULL | mysql | root[root] | localhost | | 1.0010970000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862034; select sleep(1); | | 16 | NULL | NULL | mysql | root[root] | localhost | | 1.0012100000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862035; select sleep(1); | | 17 | NULL | NULL | mysql | root[root] | localhost | | 1.0012670000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862036; select sleep(1); | | 18 | NULL | NULL | mysql | root[root] | localhost | | 1.0007300000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862037; select sleep(1); | | 19 | NULL | NULL | mysql | root[root] | localhost | | 1.0007950000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862038; select sleep(1); | | 20 | NULL | NULL | mysql | root[root] | localhost | | 1.0010920000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862039; select sleep(1); | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ 20 rows in set (0.00 sec)
Test 39 Is Agent heavy?
Test 40 Load Test # press bash shell script > vi slow_log_sleep.sh ------------------------------------------------------------------------------------ #!/bin/bash i=0 x=120 while [ $i -ne $x ] do /mysql/bin/mysql -u root -p'rex' -e"select sleep(1);" i=$(( $i + 1 )) done ------------------------------------------------------------------------------------
Test 41 Load Test Case1. press slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 34k 99k| 0 0 | 0 0 0.3|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 1082B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 300B 592B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 332B 1086B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 332B 1032B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M
Test 42 Load Test Case1. press slow query * Pressing status - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 99 0 0 0| 32k 0 | 120B 782B|1.0 0 2.0|1114M 934M| 741M 7392k 92.7M 148M 0 1 99 0 0 0| 32k 28k| 120B 1438B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 1 97 0 1 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 0 99 0 0 0| 96k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 0 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 1 98 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 1 1 97 0 0 1| 64k 0 | 120B 782B|4.0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 1 96 3 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 64k 12k| 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 96 2 0 0| 64k 0 | 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 0 100 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 98 0 0 0| 32k 0 | 210B 836B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 32k 16k| 210B 836B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 798B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 1 1 93 5 0 0| 64k 0 | 180B 824B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 1 94 5 0 0| 96k 0 | 120B 782B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M
Test 43 Load Test Case2. press fluentd & slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 1242B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 1 98 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 212B 484B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M
Test 44 Load Test Case2. press fluentd & slow query * service td-agent start - cpu : idle 99~100% - mem : used 804M (60M used) - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 32k 95k| 0 0 | 0 0 0.3|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 1082B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 12 24 4 56 2 1|2692k 0 | 226B 670B| 0 1.0 24|1112M 936M| 766M 8004k 95.5M 120M <-- agent start 19 23 0 57 0 1|3444k 0 | 60B 628B|1.0 0 0|1112M 936M| 776M 8856k 98.0M 106M 7 54 0 36 1 1| 872k 6532k| 180B 2072B|1.0 0 9.0|1118M 930M| 827M 6988k 90.6M 64.6M 11 14 51 24 0 0|2448k 48k|1921B 5230B| 0 0 9.0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 458B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 240B 550B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M
Test 45 Load Test Case2. press fluentd & slow query * Pressing status - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 98 1 0 0| 32k 24k| 180B 1112B|1.0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.8M 0 1 97 2 0 0| 64k 24k| 240B 1810B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 97 1 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.7M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.4M 0 0 100 0 0 0| 64k 0 | 180B 1218B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 96 2 0 0| 32k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 0 0 100 0 0 0| 32k 0 | 180B 1128B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 1 1 98 0 0 0| 64k 0 | 818B 2587B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 1 99 0 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.2M 1 1 98 0 0 0| 64k 0 | 180B 1112B|1.0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 0 99 1 0 0| 64k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 1 96 2 0 0| 64k 4096B| 180B 1128B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 0 1 93 6 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 0 98 1 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 0 1 96 3 0 0| 64k 32k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7384k 84.9M 88.8M 1 1 96 2 0 0| 64k 8192B| 360B 1236B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.8M 0 1 95 2 1 1| 64k 0 | 818B 2603B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 1 0 99 0 0 0| 0 0 | 210B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 0 1 99 0 0 0| 32k 0 | 270B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M
Test 46 Load Test ### Conclusion * Case1 : no fluentd agent press - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K * Case2 : Fluentd agent press - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K >>> the load is not at all due to the agent. (only 60M) * 800,000 p/s logging
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 47
To do 48 • MySQL : slow query log • MySQL : process list • MySQL : show global status • Apache : web log • Linux : Syslog • Linux : df command • Linux : dstat command . . . . .
To do 49 • 1,500 persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement 3nd Development ???
To do 50
To do 51 master ETL slave master slave master slave master slave
52 MySQL : ETL <source> type mysql_query host rex port 3306 database test username root password rex interval 10s tag rex.log_game_play query select * from test.log_game_play where log_dat e between date_sub(now(), interval 10 minute) and now (); record_hostname yes nest_result no nest_key data </source> <match rex.log_game_play> type copy <store> type stdout </store> <store> type mysql_bulk host rex port 3306 database test username root password rex column_names log_date,useridx,play_time,char_type,result key_names log_date,useridx,play_time,char_type,result table log_game_play on_duplicate_key_update true on_duplicate_update_keys log_date,useridx flush_interval 15s </store> </match> To do
To do 53 MySS 6S or MySS 7
I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 54
Reference 55 http://www.fluentd.org/casestudy/line
Reference 56 # RubyConf 2014 Build the Unified Logging Layer with Fluentd and Ruby by Kiyoto Tamura https://www.youtube.com/watch?v=sIVGsQgMHIo # Fluentd로 apache log를 MongoDB에 저장하기 http://www.sjune.net/archives/1164?ckattempt=1 # Big Data入門に見せかけたFluentd入門 http://www.slideshare.net/keithseahus/big-datafluentd # fluentd Plugins List http://www.fluentd.org/plugins/all http://www.fluentd.org/plugins https://github.com/search?utf8=✓&q=fluent-plugin+parser&type=Repositories&ref=searchresults # Fluentd - Pluggable log collector http://blog.seulgi.kim/2014/04/fluentd-pluggable-log-collector.html # Fluentd를 이용하여 모니터링도구 만들기 http://www.smallake.kr/?p=19241 # kibana https://www.elastic.co/products/kibana
thank you

Recommended

Replacing Squid with ATS
Replacing Squid with ATSReplacing Squid with ATS
Replacing Squid with ATSKit Chan
 
Failsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo HomepageFailsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo HomepageKit Chan
 
glance replicator
glance replicatorglance replicator
glance replicatoririx_jp
 
ProxySQL & PXC(Query routing and Failover Test)
ProxySQL & PXC(Query routing and Failover Test)ProxySQL & PXC(Query routing and Failover Test)
ProxySQL & PXC(Query routing and Failover Test)YoungHeon (Roy) Kim
 
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Wei Lin
 
MySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKMySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKYoungHeon (Roy) Kim
 
Query logging with proxysql
Query logging with proxysqlQuery logging with proxysql
Query logging with proxysqlYoungHeon (Roy) Kim
 
My sql failover test using orchestrator
My sql failover test using orchestratorMy sql failover test using orchestrator
My sql failover test using orchestratorYoungHeon (Roy) Kim
 

Recommended

Replacing Squid with ATS
Replacing Squid with ATSReplacing Squid with ATS
Replacing Squid with ATSKit Chan
 
Failsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo HomepageFailsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo HomepageKit Chan
 
glance replicator
glance replicatorglance replicator
glance replicatoririx_jp
 
ProxySQL & PXC(Query routing and Failover Test)
ProxySQL & PXC(Query routing and Failover Test)ProxySQL & PXC(Query routing and Failover Test)
ProxySQL & PXC(Query routing and Failover Test)YoungHeon (Roy) Kim
 
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Wei Lin
 
MySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKMySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKYoungHeon (Roy) Kim
 
Query logging with proxysql
Query logging with proxysqlQuery logging with proxysql
Query logging with proxysqlYoungHeon (Roy) Kim
 
My sql failover test using orchestrator
My sql failover test using orchestratorMy sql failover test using orchestrator
My sql failover test using orchestratorYoungHeon (Roy) Kim
 
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021Altinity Ltd
 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupDATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupSaewoong Lee
 
VCL template abstraction model and automated deployments to Fastly
VCL template abstraction model and automated deployments to FastlyVCL template abstraction model and automated deployments to Fastly
VCL template abstraction model and automated deployments to FastlyFastly
 
Script it
Script itScript it
Script itGiuseppe Maxia
 
Using ngx_lua in UPYUN
Using ngx_lua in UPYUNUsing ngx_lua in UPYUN
Using ngx_lua in UPYUNCong Zhang
 
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013Puppet
 
Replication skeptic
Replication skepticReplication skeptic
Replication skepticGiuseppe Maxia
 
[오픈소스컨설팅] Linux Network Troubleshooting
[오픈소스컨설팅] Linux Network Troubleshooting[오픈소스컨설팅] Linux Network Troubleshooting
[오픈소스컨설팅] Linux Network TroubleshootingOpen Source Consulting
 
Installation Openstack Swift
Installation Openstack SwiftInstallation Openstack Swift
Installation Openstack Swiftymtech
 
Functional Operations (Functional Programming at Comcast Labs Connect)
Functional Operations (Functional Programming at Comcast Labs Connect)Functional Operations (Functional Programming at Comcast Labs Connect)
Functional Operations (Functional Programming at Comcast Labs Connect)Susan Potter
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4琛琳 饶
 
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...Docker, Inc.
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)Altinity Ltd
 
SQL Track: Restoring databases with powershell
SQL Track: Restoring databases with powershellSQL Track: Restoring databases with powershell
SQL Track: Restoring databases with powershellITProceed
 
Dynamo: Not Just For Datastores
Dynamo: Not Just For DatastoresDynamo: Not Just For Datastores
Dynamo: Not Just For DatastoresSusan Potter
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com琛琳 饶
 
Roll Your Own API Management Platform with nginx and Lua
Roll Your Own API Management Platform with nginx and LuaRoll Your Own API Management Platform with nginx and Lua
Roll Your Own API Management Platform with nginx and LuaJon Moore
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talkLocaweb
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in VaultGlynnForrest
 
Puppet and Apache CloudStack
Puppet and Apache CloudStackPuppet and Apache CloudStack
Puppet and Apache CloudStackPuppet
 
Curso de MySQL 5.7
Curso de MySQL 5.7Curso de MySQL 5.7
Curso de MySQL 5.7Eduardo Legatti
 
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenOSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenNETWAYS
 

More Related Content

What's hot

Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021Altinity Ltd
 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupDATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupSaewoong Lee
 
VCL template abstraction model and automated deployments to Fastly
VCL template abstraction model and automated deployments to FastlyVCL template abstraction model and automated deployments to Fastly
VCL template abstraction model and automated deployments to FastlyFastly
 
Using ngx_lua in UPYUN
Using ngx_lua in UPYUNUsing ngx_lua in UPYUN
Using ngx_lua in UPYUNCong Zhang
 
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013Puppet
 
[오픈소스컨설팅] Linux Network Troubleshooting
[오픈소스컨설팅] Linux Network Troubleshooting[오픈소스컨설팅] Linux Network Troubleshooting
[오픈소스컨설팅] Linux Network TroubleshootingOpen Source Consulting
 
Installation Openstack Swift
Installation Openstack SwiftInstallation Openstack Swift
Installation Openstack Swiftymtech
 
Functional Operations (Functional Programming at Comcast Labs Connect)
Functional Operations (Functional Programming at Comcast Labs Connect)Functional Operations (Functional Programming at Comcast Labs Connect)
Functional Operations (Functional Programming at Comcast Labs Connect)Susan Potter
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4琛琳 饶
 
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...Docker, Inc.
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)Altinity Ltd
 
SQL Track: Restoring databases with powershell
SQL Track: Restoring databases with powershellSQL Track: Restoring databases with powershell
SQL Track: Restoring databases with powershellITProceed
 
Dynamo: Not Just For Datastores
Dynamo: Not Just For DatastoresDynamo: Not Just For Datastores
Dynamo: Not Just For DatastoresSusan Potter
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com琛琳 饶
 
Roll Your Own API Management Platform with nginx and Lua
Roll Your Own API Management Platform with nginx and LuaRoll Your Own API Management Platform with nginx and Lua
Roll Your Own API Management Platform with nginx and LuaJon Moore
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talkLocaweb
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in VaultGlynnForrest
 
Puppet and Apache CloudStack
Puppet and Apache CloudStackPuppet and Apache CloudStack
Puppet and Apache CloudStackPuppet
 

What's hot (20)

Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021
 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupDATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backup
 
VCL template abstraction model and automated deployments to Fastly
VCL template abstraction model and automated deployments to FastlyVCL template abstraction model and automated deployments to Fastly
VCL template abstraction model and automated deployments to Fastly
 
Script it
Script itScript it
Script it
 
Using ngx_lua in UPYUN
Using ngx_lua in UPYUNUsing ngx_lua in UPYUN
Using ngx_lua in UPYUN
 
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
Deploying VMware vCloud Hybrid Service with Puppet - PuppetConf 2013
 
Replication skeptic
Replication skepticReplication skeptic
Replication skeptic
 
[오픈소스컨설팅] Linux Network Troubleshooting
[오픈소스컨설팅] Linux Network Troubleshooting[오픈소스컨설팅] Linux Network Troubleshooting
[오픈소스컨설팅] Linux Network Troubleshooting
 
Installation Openstack Swift
Installation Openstack SwiftInstallation Openstack Swift
Installation Openstack Swift
 
Functional Operations (Functional Programming at Comcast Labs Connect)
Functional Operations (Functional Programming at Comcast Labs Connect)Functional Operations (Functional Programming at Comcast Labs Connect)
Functional Operations (Functional Programming at Comcast Labs Connect)
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4
 
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
 
SQL Track: Restoring databases with powershell
SQL Track: Restoring databases with powershellSQL Track: Restoring databases with powershell
SQL Track: Restoring databases with powershell
 
Dynamo: Not Just For Datastores
Dynamo: Not Just For DatastoresDynamo: Not Just For Datastores
Dynamo: Not Just For Datastores
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com
 
Roll Your Own API Management Platform with nginx and Lua
Roll Your Own API Management Platform with nginx and LuaRoll Your Own API Management Platform with nginx and Lua
Roll Your Own API Management Platform with nginx and Lua
 
Lua tech talk
Lua tech talkLua tech talk
Lua tech talk
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
 
Puppet and Apache CloudStack
Puppet and Apache CloudStackPuppet and Apache CloudStack
Puppet and Apache CloudStack
 

Similar to Fluentd 20150918 no_demo_public

OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenOSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenNETWAYS
 
Basic Knowledge on MySql Replication
Basic Knowledge on MySql ReplicationBasic Knowledge on MySql Replication
Basic Knowledge on MySql ReplicationTasawr Interactive
 
Undelete (and more) rows from the binary log
Undelete (and more) rows from the binary logUndelete (and more) rows from the binary log
Undelete (and more) rows from the binary logFrederic Descamps
 
MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527Saewoong Lee
 
The MySQL SYS Schema
The MySQL SYS SchemaThe MySQL SYS Schema
The MySQL SYS SchemaMark Leith
 
DB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQLDB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQLMarcelo Altmann
 
Capturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQLCapturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQLRonald Bradford
 
Capturing, Analyzing, and Optimizing your SQL
Capturing, Analyzing, and Optimizing your SQLCapturing, Analyzing, and Optimizing your SQL
Capturing, Analyzing, and Optimizing your SQLPadraig O'Sullivan
 
Windows Server 2008 (PowerShell Scripting Uygulamaları)
Windows Server 2008 (PowerShell Scripting Uygulamaları)Windows Server 2008 (PowerShell Scripting Uygulamaları)
Windows Server 2008 (PowerShell Scripting Uygulamaları)ÇözümPARK
 
Dave Williams - Nagios Log Server - Practical Experience
Dave Williams - Nagios Log Server - Practical ExperienceDave Williams - Nagios Log Server - Practical Experience
Dave Williams - Nagios Log Server - Practical ExperienceNagios
 
MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 MinutesSveta Smirnova
 
OpenStack API's and WSGI
OpenStack API's and WSGIOpenStack API's and WSGI
OpenStack API's and WSGIMike Pittaro
 
Build your own private openstack cloud
Build your own private openstack cloudBuild your own private openstack cloud
Build your own private openstack cloudNUTC, imac
 
MySQL Idiosyncrasies That Bite
MySQL Idiosyncrasies That BiteMySQL Idiosyncrasies That Bite
MySQL Idiosyncrasies That BiteRonald Bradford
 
Mysql nowwhat
Mysql nowwhatMysql nowwhat
Mysql nowwhatsqlhjalp
 

Similar to Fluentd 20150918 no_demo_public (20)

Curso de MySQL 5.7
Curso de MySQL 5.7Curso de MySQL 5.7
Curso de MySQL 5.7
 
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenOSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
 
Mysql56 replication
Mysql56 replicationMysql56 replication
Mysql56 replication
 
Basic Knowledge on MySql Replication
Basic Knowledge on MySql ReplicationBasic Knowledge on MySql Replication
Basic Knowledge on MySql Replication
 
MySQLinsanity
MySQLinsanityMySQLinsanity
MySQLinsanity
 
Undelete (and more) rows from the binary log
Undelete (and more) rows from the binary logUndelete (and more) rows from the binary log
Undelete (and more) rows from the binary log
 
MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527
 
The MySQL SYS Schema
The MySQL SYS SchemaThe MySQL SYS Schema
The MySQL SYS Schema
 
DB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQLDB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQL
 
Capturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQLCapturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQL
 
Capturing, Analyzing, and Optimizing your SQL
Capturing, Analyzing, and Optimizing your SQLCapturing, Analyzing, and Optimizing your SQL
Capturing, Analyzing, and Optimizing your SQL
 
Windows Server 2008 (PowerShell Scripting Uygulamaları)
Windows Server 2008 (PowerShell Scripting Uygulamaları)Windows Server 2008 (PowerShell Scripting Uygulamaları)
Windows Server 2008 (PowerShell Scripting Uygulamaları)
 
Dave Williams - Nagios Log Server - Practical Experience
Dave Williams - Nagios Log Server - Practical ExperienceDave Williams - Nagios Log Server - Practical Experience
Dave Williams - Nagios Log Server - Practical Experience
 
PHP tips by a MYSQL DBA
PHP tips by a MYSQL DBAPHP tips by a MYSQL DBA
PHP tips by a MYSQL DBA
 
MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes
 
OpenStack API's and WSGI
OpenStack API's and WSGIOpenStack API's and WSGI
OpenStack API's and WSGI
 
Build your own private openstack cloud
Build your own private openstack cloudBuild your own private openstack cloud
Build your own private openstack cloud
 
MySQL SQL Tutorial
MySQL SQL TutorialMySQL SQL Tutorial
MySQL SQL Tutorial
 
MySQL Idiosyncrasies That Bite
MySQL Idiosyncrasies That BiteMySQL Idiosyncrasies That Bite
MySQL Idiosyncrasies That Bite
 
Mysql nowwhat
Mysql nowwhatMysql nowwhat
Mysql nowwhat
 

Recently uploaded

Study on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube ExchangerStudy on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube ExchangerAnamika Sarkar
 
Call Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile serviceCall Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile servicerehmti665
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvLewisJB
 
Indian Dairy Industry Present Status and.ppt
Indian Dairy Industry Present Status and.pptIndian Dairy Industry Present Status and.ppt
Indian Dairy Industry Present Status and.pptMadan Karki
 
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)Dr SOUNDIRARAJ N
 
Risk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfRisk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfROCENODodongVILLACER
 
Transport layer issues and challenges - Guide
Transport layer issues and challenges - GuideTransport layer issues and challenges - Guide
Transport layer issues and challenges - GuideGOPINATHS437943
 
Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024hassan khalil
 
Vishratwadi & Ghorpadi Bridge Tender documents
Vishratwadi & Ghorpadi Bridge Tender documentsVishratwadi & Ghorpadi Bridge Tender documents
Vishratwadi & Ghorpadi Bridge Tender documentsSachinPawar510423
 
Call Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call GirlsCall Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call Girlsssuser7cb4ff
 
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor CatchersTechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catcherssdickerson1
 
Earthing details of Electrical Substation
Earthing details of Electrical SubstationEarthing details of Electrical Substation
Earthing details of Electrical Substationstephanwindworld
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionMebane Rash
 
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdfCCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdfAsst.prof M.Gokilavani
 
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsyncWhy does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsyncssuser2ae721
 
Arduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptArduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptSAURABHKUMAR892774
 
Application of Residue Theorem to evaluate real integrations.pptx
Application of Residue Theorem to evaluate real integrations.pptxApplication of Residue Theorem to evaluate real integrations.pptx
Application of Residue Theorem to evaluate real integrations.pptx959SahilShah
 

Recently uploaded (20)

Study on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube ExchangerStudy on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
Study on Air-Water & Water-Water Heat Exchange in a Finned Tube Exchanger
 
Call Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile serviceCall Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile service
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvv
 
Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptx
Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptxExploring_Network_Security_with_JA3_by_Rakesh Seal.pptx
Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptx
 
Design and analysis of solar grass cutter.pdf
Design and analysis of solar grass cutter.pdfDesign and analysis of solar grass cutter.pdf
Design and analysis of solar grass cutter.pdf
 
young call girls in Rajiv Chowk🔝 9953056974 🔝 Delhi escort Service
young call girls in Rajiv Chowk🔝 9953056974 🔝 Delhi escort Serviceyoung call girls in Rajiv Chowk🔝 9953056974 🔝 Delhi escort Service
young call girls in Rajiv Chowk🔝 9953056974 🔝 Delhi escort Service
 
Indian Dairy Industry Present Status and.ppt
Indian Dairy Industry Present Status and.pptIndian Dairy Industry Present Status and.ppt
Indian Dairy Industry Present Status and.ppt
 
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
 
Risk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfRisk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdf
 
Transport layer issues and challenges - Guide
Transport layer issues and challenges - GuideTransport layer issues and challenges - Guide
Transport layer issues and challenges - Guide
 
Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024
 
Vishratwadi & Ghorpadi Bridge Tender documents
Vishratwadi & Ghorpadi Bridge Tender documentsVishratwadi & Ghorpadi Bridge Tender documents
Vishratwadi & Ghorpadi Bridge Tender documents
 
Call Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call GirlsCall Girls Narol 7397865700 Independent Call Girls
Call Girls Narol 7397865700 Independent Call Girls
 
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor CatchersTechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
 
Earthing details of Electrical Substation
Earthing details of Electrical SubstationEarthing details of Electrical Substation
Earthing details of Electrical Substation
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of Action
 
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdfCCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf
 
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsyncWhy does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
 
Arduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptArduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.ppt
 
Application of Residue Theorem to evaluate real integrations.pptx
Application of Residue Theorem to evaluate real integrations.pptxApplication of Residue Theorem to evaluate real integrations.pptx
Application of Residue Theorem to evaluate real integrations.pptx
 

Fluentd 20150918 no_demo_public

  • 2. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 2
  • 3. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 3
  • 4. Intro 4 • 1,500 persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement MySS architecture
  • 5. Intro 5 • 1,500 persons • 20,000,000,000 yen 3nd Development ??? (real time slow query)
  • 6. Intro 6 • 1,500 persons • 20,000,000,000 yen
  • 7. Intro 7 Before fluentd Duplicated codes for error logic Complex codes for retry logic
  • 9. Intro 9 • 1,500 persons • 20,000,000,000 yen Scribe Flume Fluentd logstash
  • 10. Intro 10 구분 강조점 유연성 / 상호운용성 단순함 / 안정성 구현언어 J 루비(Jruby) C루비(Cruby) 입력방법 많음 적음 플러그인 많음 매우 많음 플러그인 설치 쉬움 (jar 배포) 쉬움 (gem 설치) 신뢰성 메시지 loss 가능성 있음 메시지 loss 없음
  • 11. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 11
  • 13. Intro 13 Mysql : slow query Input Output +--------------------------------------------+ | sql_text | +--------------------------------------------+ | SET timestamp=1441820542; select sleep(1); | | SET timestamp=1441820574; select sleep(1); | +--------------------------------------------+
  • 14. Intro 14 Mysql : slow query + LINE Input Output Parser parser Filter hostname +------------------+ | hostname | +------------------+ | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | | localhost.localdomain | +-----------------------+ +-------------+ | sql_text | +-------------+ | select sleep(1) | | select sleep(1) | | select sleep(1) | | select sleep(1) | +-------------+
  • 15. Intro 15 Mysql : process list Input Output
  • 17. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 17
  • 18. Install 18 1. Install > curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh 2. Start demon > service td-agent start Starting td-agent: [ OK ] > service td-agent status td-agent (pid 2567) is running > service td-agent restart > service td-agent stop > td-agent --version td-agent 0.12.12 3. Transfer Sample Log > curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test.test > tail -n 1 /var/log/td-agent/td-agent.log 2015-09-10 00:26:31 +0900 debug.test.test: {"json":"message"}
  • 19. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 19
  • 20. Test 20 MySQL : slow query <source> type mysql_slow_query path /mysql/logs/slow_query.log pos_file /mysql/logs/slow_query.pos tag raw.collect.mysql_slow </source> <match raw.collect.mysql_slow> type parser remove_prefix raw format /^SET timestamp=(?<time_stamp>[^;]*); (?<sql>[^;]*);$/ time_format %d/%b/%Y:%H:%M:%S %z key_name sql reserve_data yes </match> <match collect.mysql_slow> type hostname key_name hostname add_prefix filtered </match> <match filtered.collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql_text key_names hostname,user,host,db,query_time,lock_time,rows_sent,rows_examined,ti me_stamp,sql table t_mysql_slow flush_interval 5s </store> </match>
  • 21. Test 21 MySQL : slow query * Tail Log * MySQL Data
  • 22. Test 22 MySQL : slow query # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_slow; create table DBPart.t_mysql_slow ( seq bigint auto_increment , hostname varchar(30) , log_date datetime , db varchar(100) , user varchar(100) , host varchar(100) , host_ip varchar(20) , query_time decimal(20,10) , lock_time decimal(20,10) , rows_sent bigint , rows_examined bigint , sql_text varchar(10000) , primary key(seq) , key(log_date) );
  • 23. Test 23 MySQL : slow query # Client 1. fluentd plugin install * fluent-plugin-mysqlslowquery / fluent-plugin-mysql / fluent-plugin-mysql-bulk https://github.com/yuku-t/fluent-plugin-mysqlslowquery https://github.com/tagomoris/fluent-plugin-mysql https://github.com/toyama0919/fluent-plugin-mysql-bulk > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysqlslowquery > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-bulk 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
  • 24. Test 24 MySQL : slow query > vi /etc/td-agent/td-agent.conf ========================================================= ... <source> type mysql_slow_query path /mysql/logs/slow_query.log tag collect.mysql_slow </source> <match collect.mysql_slow> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.100 port 3306 database DBPart username root password rex column_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql_text key_names user,host,host_ip,query_time,lock_time,rows_sent,rows_examined,sql table t_mysql_slow flush_interval 5s </store> </match> =========================================================
  • 25. Test 25 MySQL : slow query 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. press slow query mysql -u root -p use mysql; select sleep(1); select sleep(1); [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log 2015-09-10 12:06:31 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000855,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854391; select sleep(1);","db":"mysql"} 2015-09-10 12:06:33 +0900 collect.mysql_slow: {"user":"root[root]","host":"localhost","host_ip":"","query_time":1.000862,"lock_time":0.0,"rows_sent":1,"rows_examine d":0,"sql":"SET timestamp=1441854393; select sleep(1);","db":"mysql"} 2015-09-10 12:24:16 +0900 [info]: bulk insert values size => 2 * SERVER mysql> select * from t_mysql_slow; +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | log_date | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | hostname | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ | 2015-09-10 07:03:45 | root[root] | localhost | | 1.0008600000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820542; select sleep(1); | NULL | | 2015-09-10 07:04:15 | root[root] | localhost | | 1.0010030000 | 0.0000000000 | 1 | 0 | SET timestamp=1441820574; select sleep(1); | NULL | +---------------------+------------+-----------+---------+--------------+--------------+-----------+---------------+--------------------------------------------+----------+ 8 rows in set (0.00 sec)
  • 26. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 26
  • 27. Test 27 MySQL : show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duration_time key_names hostname,Id,User,Host,db,Command,State,Info,duration_time table t_mysql_process flush_interval 5s </store> </match>
  • 28. Test 28 * Tail Log * MySQL Data MySQL : show processlist
  • 29. Test 29 MySQL : show processlist # Server 1. Create Log table in Collector Server mysql -u root -p use DBPart; drop table if exists DBPart.t_mysql_process; create table DBPart.t_mysql_process ( hostname varchar(100) , log_date datetime default current_timestamp , id bigint , user varchar(100) , host varchar(100) , db varchar(64) , command varchar(50) , duration_time bigint , state varchar(4000) , info varchar(10000) );
  • 30. Test 30 MySQL : show processlist # Client 1. fluentd plugin install * fluent-plugin-mysql-query / fluent-plugin-rename-key https://github.com/y-ken/fluent-plugin-mysql-query https://github.com/shunwen/fluent-plugin-rename-key > yum -y install ruby-rdoc ruby-devel rubygems > find / -name fluent-gem /opt/td-agent/embedded/bin/fluent-gem /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluent-gem > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-mysql-query > /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-rename-key 2. config > vi /etc/init.d/td-agent #TD_AGENT_USER=td-agent TD_AGENT_USER=root
  • 31. Test 31 MySQL : show processlist > vi /etc/td-agent/td-agent.conf ========================================= ... ### mysql show processlist <source> type mysql_query host 192.168.95.222 port 3306 database DBPart username root password rex interval 10s tag client.processlist query show full processlist; record_hostname yes nest_result no nest_key data #row_count yes #row_count_key row_count </source> <match client.processlist> type rename_key remove_tag_prefix client. append_tag client rename_rule1 Time duration_time </match> <match processlist.client> type copy <store> type stdout </store> <store> type mysql_bulk host 192.168.95.57 port 3306 database DBPart username root password rex column_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time key_names hostname,Id,User,Host,db,Command,State,Info,duratio n_time table t_mysql_process flush_interval 5s </store> </match> =========================================
  • 32. Test 32 MySQL : show processlist 3. td-agent restart service td-agent stop Shutting down td-agent: [ OK ] service td-agent start Starting td-agent: [ OK ] 4. show processlist; * client mysql -u root -p root@localhost:(none) 14:43:47>show processlist; +----+------+----------------------+--------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+----------------------+--------+---------+------+-------+------------------+ | 3 | root | localhost | NULL | Sleep | 381 | | NULL | | 7 | root | localhost | NULL | Query | 0 | init | show processlist | | 9 | root | 192.168.95.222:59359 | DBPart | Sleep | 43 | | NULL | | 10 | root | localhost | NULL | Sleep | 3 | | NULL | +----+------+----------------------+--------+---------+------+-------+------------------+ 4 rows in set (0.00 sec) [root@localhost ~]# tail -f /var/log/td-agent/td-agent.log … 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"3","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"578","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"7","User":"root","Host":"localhost","db":null,"Command":"Sleep","duration_time":"60","State":"","Info":null} 2015-09-10 14:47:29 +0900 processlist.collect: {"hostname":"localhost.localdomain","Id":"9","User":"root","Host":"192.168.95.222:59359","db":"DBPart","Command":"Query","duration_time":"0","State":"init","Info":"show full processlist"} 2015-09-10 14:47:30 +0900 [info]: bulk insert values size => 3 --> Send every 1 minutes
  • 33. Test 33 MySQL : show processlist * SERVER root@localhost:DBPart 15:07:51>select * from t_mysql_process; +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | log_date | hostname | id | user | host | db | command | duration_time | state | info | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ | 2015-09-10 15:04:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 338 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 25 | | NULL | | 2015-09-10 15:04:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 398 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 17 | | NULL | | 2015-09-10 15:05:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:05:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 20 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 458 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 16 | | NULL | | 2015-09-10 15:06:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | | 2015-09-10 15:06:49 | localhost.localdomain | 10 | root | localhost | NULL | Sleep | 80 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 3 | root | localhost | NULL | Sleep | 518 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 7 | root | localhost | NULL | Sleep | 7 | | NULL | | 2015-09-10 15:07:49 | localhost.localdomain | 9 | root | 192.168.95.222:59359 | DBPart | Query | 0 | init | show full processlist | +---------------------+-----------------------+------+------+----------------------+--------+---------+---------------+-------+-----------------------+ 14 rows in set (0.00 sec)
  • 36. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Error / Load test V. To do VI. Reference INDEX Agenda 36
  • 37. Test 37 If client server (mysql) was down?
  • 38. Test 38 Error Test Case. MySQL server down Error * client use mysql; select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server stop select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); select sleep(1); <-- mysql server start * server > select * from t_mysql_slow; +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | seq | hostname | log_date | db | user | host | host_ip | query_time | lock_time | rows_sent | rows_examined | sql_text | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ | 1 | NULL | NULL | mysql | root[root] | localhost | | 1.0015090000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862020; select sleep(1); | | 2 | NULL | NULL | mysql | root[root] | localhost | | 1.0004050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862021; select sleep(1); | | 3 | NULL | NULL | mysql | root[root] | localhost | | 1.0004840000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862022; select sleep(1); | | 4 | NULL | NULL | mysql | root[root] | localhost | | 1.0004360000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862023; select sleep(1); | | 5 | NULL | NULL | mysql | root[root] | localhost | | 1.0011230000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862024; select sleep(1); | | 6 | NULL | NULL | mysql | root[root] | localhost | | 1.0007080000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862025; select sleep(1); | | 7 | NULL | NULL | mysql | root[root] | localhost | | 1.0012720000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862026; select sleep(1); | | 8 | NULL | NULL | mysql | root[root] | localhost | | 1.0004110000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862027; select sleep(1); | | 9 | NULL | NULL | mysql | root[root] | localhost | | 1.0003540000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862028; select sleep(1); | | 10 | NULL | NULL | mysql | root[root] | localhost | | 1.0003870000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862029; select sleep(1); | | 11 | NULL | NULL | mysql | root[root] | localhost | | 1.0005050000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862030; select sleep(1); | | 12 | NULL | NULL | mysql | root[root] | localhost | | 1.0005450000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862031; select sleep(1); | | 13 | NULL | NULL | mysql | root[root] | localhost | | 1.0002890000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862032; select sleep(1); | | 14 | NULL | NULL | mysql | root[root] | localhost | | 1.0011930000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862033; select sleep(1); | | 15 | NULL | NULL | mysql | root[root] | localhost | | 1.0010970000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862034; select sleep(1); | | 16 | NULL | NULL | mysql | root[root] | localhost | | 1.0012100000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862035; select sleep(1); | | 17 | NULL | NULL | mysql | root[root] | localhost | | 1.0012670000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862036; select sleep(1); | | 18 | NULL | NULL | mysql | root[root] | localhost | | 1.0007300000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862037; select sleep(1); | | 19 | NULL | NULL | mysql | root[root] | localhost | | 1.0007950000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862038; select sleep(1); | | 20 | NULL | NULL | mysql | root[root] | localhost | | 1.0010920000 | 0.0000000000 | 1 | 0 | SET timestamp=1441862039; select sleep(1); | +-----+----------+----------+-------+------------+-----------+---------+--------------+--------------+-----------+------------ ---+--------------------------------------------+ 20 rows in set (0.00 sec)
  • 40. Test 40 Load Test # press bash shell script > vi slow_log_sleep.sh ------------------------------------------------------------------------------------ #!/bin/bash i=0 x=120 while [ $i -ne $x ] do /mysql/bin/mysql -u root -p'rex' -e"select sleep(1);" i=$(( $i + 1 )) done ------------------------------------------------------------------------------------
  • 41. Test 41 Load Test Case1. press slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 34k 99k| 0 0 | 0 0 0.3|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 1082B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 300B 592B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 332B 1086B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 332B 1032B| 0 0 1.0|1118M 930M| 737M 7304k 92.7M 152M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 99 0 0 1| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 1 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 737M 7304k 92.7M 152M
  • 42. Test 42 Load Test Case1. press slow query * Pressing status - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 99 0 0 0| 32k 0 | 120B 782B|1.0 0 2.0|1114M 934M| 741M 7392k 92.7M 148M 0 1 99 0 0 0| 32k 28k| 120B 1438B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 1 97 0 1 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 1 0 99 0 0 0| 96k 0 | 120B 782B| 0 0 2.0|1114M 934M| 741M 7400k 92.7M 148M 0 1 99 0 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 0 99 0 0 0| 64k 0 | 120B 798B| 0 0 2.0|1114M 934M| 741M 7408k 92.7M 148M 1 1 98 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 1 1 97 0 0 1| 64k 0 | 120B 782B|4.0 0 2.0|1114M 934M| 742M 7408k 92.7M 147M 0 1 96 3 0 0| 32k 16k| 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 64k 12k| 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 96 2 0 0| 64k 0 | 210B 852B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 0 100 0 0 0| 64k 0 | 120B 782B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 1 1 98 0 0 0| 32k 0 | 210B 836B| 0 0 2.0|1114M 934M| 742M 7416k 92.7M 147M 0 1 99 0 0 0| 32k 16k| 210B 836B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 0 100 0 0 0| 32k 0 | 120B 798B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 1 1 93 5 0 0| 64k 0 | 180B 824B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M 0 1 94 5 0 0| 96k 0 | 120B 782B| 0 0 2.0|1113M 935M| 742M 7424k 92.7M 147M
  • 43. Test 43 Load Test Case2. press fluentd & slow query * Before press status - cpu : idle 99~100% - mem : used 737M - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 1242B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 1 98 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 212B 484B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 1 99 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 242B 496B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 244B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 152B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M
  • 44. Test 44 Load Test Case2. press fluentd & slow query * service td-agent start - cpu : idle 99~100% - mem : used 804M (60M used) - I/O : read 0K, write 0K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 0 0 99 1 0 0| 32k 95k| 0 0 | 0 0 0.3|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 1082B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 1 0 99 0 0 0| 0 0 | 60B 442B| 0 0 0|1112M 936M| 743M 7652k 93.2M 145M 12 24 4 56 2 1|2692k 0 | 226B 670B| 0 1.0 24|1112M 936M| 766M 8004k 95.5M 120M <-- agent start 19 23 0 57 0 1|3444k 0 | 60B 628B|1.0 0 0|1112M 936M| 776M 8856k 98.0M 106M 7 54 0 36 1 1| 872k 6532k| 180B 2072B|1.0 0 9.0|1118M 930M| 827M 6988k 90.6M 64.6M 11 14 51 24 0 0|2448k 48k|1921B 5230B| 0 0 9.0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 458B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7244k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 150B 496B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 240B 550B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M 0 0 100 0 0 0| 0 0 | 60B 442B| 0 0 0|1118M 930M| 804M 7252k 84.8M 93.2M
  • 45. Test 45 Load Test Case2. press fluentd & slow query * Pressing status - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K [root@localhost ~]# dstat -c -d -n -p -s -m ----total-cpu-usage---- -dsk/total- -net/total- ---procs--- ----swap--- ------memory-usage----- usr sys idl wai hiq siq| read writ| recv send|run blk new| used free| used buff cach free 1 0 98 1 0 0| 32k 24k| 180B 1112B|1.0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.8M 0 1 97 2 0 0| 64k 24k| 240B 1810B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 97 1 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.7M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.4M 0 0 100 0 0 0| 64k 0 | 180B 1218B| 0 0 2.0|1115M 933M| 807M 7360k 84.9M 89.6M 1 1 96 2 0 0| 32k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 0 0 100 0 0 0| 32k 0 | 180B 1128B| 0 0 2.0|1115M 933M| 807M 7368k 84.9M 89.4M 1 1 98 0 0 0| 64k 0 | 818B 2587B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 1 99 0 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.2M 1 1 98 0 0 0| 64k 0 | 180B 1112B|1.0 0 2.0|1115M 933M| 808M 7368k 84.9M 89.3M 0 0 99 1 0 0| 64k 16k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 1 96 2 0 0| 64k 4096B| 180B 1128B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 0 1 93 6 0 0| 64k 0 | 818B 2602B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.1M 1 0 98 1 0 0| 32k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 1 1 95 3 0 0| 64k 0 | 180B 1112B| 0 0 2.0|1115M 933M| 808M 7376k 84.9M 89.0M 0 1 96 3 0 0| 64k 32k| 180B 1112B| 0 0 2.0|1115M 933M| 808M 7384k 84.9M 88.8M 1 1 96 2 0 0| 64k 8192B| 360B 1236B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.8M 0 1 95 2 1 1| 64k 0 | 818B 2603B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 1 0 99 0 0 0| 0 0 | 210B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M 0 1 99 0 0 0| 32k 0 | 270B 1166B| 0 0 2.0|1114M 934M| 808M 7384k 84.9M 88.7M
  • 46. Test 46 Load Test ### Conclusion * Case1 : no fluentd agent press - cpu : idle 94~100% (used +5%) - mem : used 741M (used +4M) - I/O : read 32K~64K, write 12K~32K * Case2 : Fluentd agent press - cpu : idle 95~100% (used +4%) - mem : used 809M (used +4M) - I/O : read 32K~64K, write 16K~36K >>> the load is not at all due to the agent. (only 60M) * 800,000 p/s logging
  • 47. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 47
  • 48. To do 48 • MySQL : slow query log • MySQL : process list • MySQL : show global status • Apache : web log • Linux : Syslog • Linux : df command • Linux : dstat command . . . . .
  • 49. To do 49 • 1,500 persons • 20,000,000,000 yen slow_query Collect DB WEB VIEWER 1st Devleopement 2nd Devleopement 3nd Devleopement 3nd Development ???
  • 51. To do 51 master ETL slave master slave master slave master slave
  • 52. 52 MySQL : ETL <source> type mysql_query host rex port 3306 database test username root password rex interval 10s tag rex.log_game_play query select * from test.log_game_play where log_dat e between date_sub(now(), interval 10 minute) and now (); record_hostname yes nest_result no nest_key data </source> <match rex.log_game_play> type copy <store> type stdout </store> <store> type mysql_bulk host rex port 3306 database test username root password rex column_names log_date,useridx,play_time,char_type,result key_names log_date,useridx,play_time,char_type,result table log_game_play on_duplicate_key_update true on_duplicate_update_keys log_date,useridx flush_interval 15s </store> </match> To do
  • 53. To do 53 MySS 6S or MySS 7
  • 54. I. Intro II. Architecture III. Install IV. Test - MySQL slowquery logging - MySQL processlist logging - Load test V. To do VI. Reference INDEX Agenda 54
  • 56. Reference 56 # RubyConf 2014 Build the Unified Logging Layer with Fluentd and Ruby by Kiyoto Tamura https://www.youtube.com/watch?v=sIVGsQgMHIo # Fluentd로 apache log를 MongoDB에 저장하기 http://www.sjune.net/archives/1164?ckattempt=1 # Big Data入門に見せかけたFluentd入門 http://www.slideshare.net/keithseahus/big-datafluentd # fluentd Plugins List http://www.fluentd.org/plugins/all http://www.fluentd.org/plugins https://github.com/search?utf8=✓&q=fluent-plugin+parser&type=Repositories&ref=searchresults # Fluentd - Pluggable log collector http://blog.seulgi.kim/2014/04/fluentd-pluggable-log-collector.html # Fluentd를 이용하여 모니터링도구 만들기 http://www.smallake.kr/?p=19241 # kibana https://www.elastic.co/products/kibana