ソーシャルアプリ向けシステム監視運用の勘所

Tatsuro Hisamori DeNA Co.,Ltd. 2010 10 17

• (Tatsuro Hisamori) • • mbga OpenPlatform • / • • etc 2010 10 17

mbga OpenPlatform • 16:00 id:ZIGOROu Inside mbga Platform (^0^v 2010 10 17

• Social Application / 2010 10 17

• • 2010 10 17

2010 10 17

• • • Gadget / API • • Platform • • tcpdump 2010 10 17

• • • DB • 2010 10 17

• • • DB • • • 2010 10 17

• request -> response • WAF • request ﬁnalize • Apache • CustomLog %D 2010 10 17

use Time::HiRes qw/gettimeofday tv_interval/; sub pre_request { my ( $c, $args ) = @_; $c->stash->{__start_time} = [gettimeofday]; } sub ﬁnalize { my ( $c, $args ) = @_; $c->log->debug( $process_time = tv_interval($c->stash->{__start_time}, [gettimeofday])); } 2010 10 17

• %D GadgetServer • %D request line response # protocol.c (httpd-2.2.16) static int read_request_line(request_rec *r, apr_bucket_brigade *bb) { r->request_time = apr_time_now(); } 2010 10 17

• GadgetServer • alarm() SIGALRM # eval { local $SIG{ALRM} = sub {die}; alarm($TIMEOUT); $response = $ua->request($URL); alarm(0); } ) 2010 10 17

• TCP • tcpdump + wireshark Casual (^0^v 2010 10 17

$ tcpdump -i bond0 -s 1500 -n –w tcp.dump 'host gadgetserver' • -w • wireshark • stream 2010 10 17

DB • DB • DBI::Proﬁle • DBIx::ProﬁleManager • mk-query-digest(maatkit) 2010 10 17

DBI::Profile • DBI use DBI; use DBI::Profile; my $dbh = DBI->connect("dbi:mysql:db=test;”,"root", ””); $dbh->{Profile} = q|!Statement|; $dbh->selectall_arrayref("SELECT ?", undef, $_) for ( qw/1 2 3/ ); DBI::Profile: 0.000399s 91.38% (7 calls) test.pl @ 2010-10-12 21:08:03 'SELECT ?' => 0.000399s / 7 = 0.000057s avg (first 0.000177s, min 0.000002s, max 0.000177s) 2010 10 17

DBIx::ProfileManager • handle attibute use DBI; use DBIx::ProfileManager; my $dbh = DBI->connect(...); my $pm = DBIx::ProfileManager->new; $pm->profile_start; my $res = $dbh->selectall_arrayref(...); $pm->profile_stop; my @results = $pm->data_formatted; local $, = "n"; print @results; 2010 10 17

DBIx::ProfileManager • profile_start profile_stop • data_formatted format # sample $pm->data_formatted( q|%{statement} : %{max}, %{min}, %{avg}| ); • DeNA • http://engineer.dena.jp/2010/03/dbixprofilemanager-sql-profiling.html 2010 10 17

Catalyst # MyApp.pm use DBIx::ProfileManager; sub prepare_request { my ( $c, $args ) = @_; my $pm = DBIx::ProfileManager->new(%config); $pm->profile_start; $c->stash->{__dbix_profilemanager} = $pm; } sub finalize { my ( $c, $args ) = @_; my $pm = $c->stash->{__dbix_priflemanager}; $pm->profile_stop; # $pm->data_formatted } 2010 10 17

DB • slowlog mk-query-digest # Query 8: 44.67 QPS, 0.13x concurrency, ID 0x6516D7E6698E425A at byte 108467522 # Attribute pct total min max avg 95% stddev median # ========= ====== ======= ======= ======= ======= ======= ======= ======= # Count 2 317 # Exec time 2 944ms 0 205ms 3ms 2ms 22ms 108us # Hosts 181 10.5.9.20 (7/2%), 10.5.10.157 (6/1%)... 179 more # Databases 1 platform # Time range 2010-10-12 13:21:53.218711 to 2010-10-12 13:22:00.315719 # bytes 1 20.43k 66 68 66.01 65.89 0 65.89 # Errors 1 none # Rows affe 0 0 0 0 0 0 0 0 # Warning c 0 0 0 0 0 0 0 0 # Query_time distribution # 1us # 10us #################################################### # 100us ################################################################ # 1ms ######### # 10ms # # 100ms # # 1s # 10s+ # Tables # SHOW TABLE STATUS FROM `db` LIKE 'entry'G # SHOW CREATE TABLE `db`.`entry`G # EXPLAIN select id, type from entry where id = xxx G 2010 10 17

DB • # Query_time distribution # 1us # 10us #################################################### # 100us ###################################################### # 1ms ######### # 10ms # # 100ms # # 1s # 10s+ • • SQL explain 2010 10 17

• • • ↓ mk-query-digest tcpdump !! mysql dump OK 2010 10 17

DB • • • • • • 2010 10 17

DB • • $mysqladmin -uroot -p extended-STATUS | egrep '(Max|Threads_)' | Max_used_connections | 68 | | Threads_cached | 36 | | Threads_connected | 32 | | Threads_created | 68 | | Threads_running |5 | 2010 10 17

DB • • $ mysqladmin -uroot -p -hdbsrv status | awk '{print $22}' 9780.895 2010 10 17

DB • • $ sar 13:00:01 CPU %user %nice %system %iowait %idle 13:10:01 all 0.97 0.00 0.32 0.36 98.35 13:20:01 all 0.80 0.00 0.30 0.25 98.65 13:30:01 all 0.81 0.00 0.29 0.28 98.62 13:40:01 all 0.73 0.00 0.28 0.24 98.76 2010 10 17

ex. open Platform • • • • • 2010 10 17

• • from tcpdump & GadgetServer timeout • 100 200ms • ex. 2010 10 17

• • ping, traceroute etc • • 10ms ( ) • EC2 100ms 200ms 2010 10 17

• • platform <-> • <-> (Web <-> DB) • VM • I/O • 2010 10 17

• • ex. Platform @mbga OpenPlatform • TCO 2010 10 17

• • • DB • • • 2010 10 17

2010 10 17

Thanks! 2010 10 17

http://ke-tai.org	5573
http://blog.myfinder.jp	508
http://webcache.googleusercontent.com	14
http://www.iweb34.com	2
file://	1
http://www.forcepenguin.com	1
http://fr.nexthop.jp	1
http://j2k.naver.com	1
http://twittertim.es	1
http://cache.yahoofs.jp	1

ソーシャルアプリ向けシステム監視運用の勘所

by Tatsuro Hisamori on Oct 17, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

10 Embeds 6,103

Statistics

ソーシャルアプリ向けシステム監視運用の勘所 — Presentation Transcript