ソーシャルアプリ向けシステム監視運用の勘所

22,133 views

Published on

Published in: Technology
0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
22,133
On SlideShare
0
From Embeds
0
Number of Embeds
10,379
Actions
Shares
0
Downloads
16
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

ソーシャルアプリ向けシステム監視運用の勘所

  1. 1. Tatsuro Hisamori DeNA Co.,Ltd. 2010 10 17
  2. 2. • (Tatsuro Hisamori) • • mbga OpenPlatform • / • • etc 2010 10 17
  3. 3. mbga OpenPlatform • 16:00 id:ZIGOROu Inside mbga Platform (^0^v 2010 10 17
  4. 4. • Social Application / 2010 10 17
  5. 5. • • 2010 10 17
  6. 6. 2010 10 17
  7. 7. • • • Gadget / API • • Platform • • tcpdump 2010 10 17
  8. 8. • • • DB • 2010 10 17
  9. 9. • • • DB • • • 2010 10 17
  10. 10. • request -> response • WAF • request finalize • Apache • CustomLog %D 2010 10 17
  11. 11. use Time::HiRes qw/gettimeofday tv_interval/; sub pre_request { my ( $c, $args ) = @_; $c->stash->{__start_time} = [gettimeofday]; } sub finalize { my ( $c, $args ) = @_; $c->log->debug( $process_time = tv_interval($c->stash->{__start_time}, [gettimeofday])); } 2010 10 17
  12. 12. • %D GadgetServer • %D request line response # protocol.c (httpd-2.2.16) static int read_request_line(request_rec *r, apr_bucket_brigade *bb) { r->request_time = apr_time_now(); } 2010 10 17
  13. 13. • GadgetServer • alarm() SIGALRM # eval { local $SIG{ALRM} = sub {die}; alarm($TIMEOUT); $response = $ua->request($URL); alarm(0); } ) 2010 10 17
  14. 14. • TCP • tcpdump + wireshark Casual (^0^v 2010 10 17
  15. 15. $ tcpdump -i bond0 -s 1500 -n –w tcp.dump 'host gadgetserver' • -w • wireshark • stream 2010 10 17
  16. 16. DB • DB • DBI::Profile • DBIx::ProfileManager • mk-query-digest(maatkit) 2010 10 17
  17. 17. DBI::Profile • DBI use DBI; use DBI::Profile; my $dbh = DBI->connect("dbi:mysql:db=test;”,"root", ””); $dbh->{Profile} = q|!Statement|; $dbh->selectall_arrayref("SELECT ?", undef, $_) for ( qw/1 2 3/ ); DBI::Profile: 0.000399s 91.38% (7 calls) test.pl @ 2010-10-12 21:08:03 'SELECT ?' => 0.000399s / 7 = 0.000057s avg (first 0.000177s, min 0.000002s, max 0.000177s) 2010 10 17
  18. 18. DBIx::ProfileManager • handle attibute use DBI; use DBIx::ProfileManager; my $dbh = DBI->connect(...); my $pm = DBIx::ProfileManager->new; $pm->profile_start; my $res = $dbh->selectall_arrayref(...); $pm->profile_stop; my @results = $pm->data_formatted; local $, = "n"; print @results; 2010 10 17
  19. 19. DBIx::ProfileManager • profile_start profile_stop • data_formatted format # sample $pm->data_formatted( q|%{statement} : %{max}, %{min}, %{avg}| ); • DeNA • http://engineer.dena.jp/2010/03/dbixprofilemanager-sql-profiling.html 2010 10 17
  20. 20. Catalyst # MyApp.pm use DBIx::ProfileManager; sub prepare_request { my ( $c, $args ) = @_; my $pm = DBIx::ProfileManager->new(%config); $pm->profile_start; $c->stash->{__dbix_profilemanager} = $pm; } sub finalize { my ( $c, $args ) = @_; my $pm = $c->stash->{__dbix_priflemanager}; $pm->profile_stop; # $pm->data_formatted } 2010 10 17
  21. 21. DB • slowlog mk-query-digest # Query 8: 44.67 QPS, 0.13x concurrency, ID 0x6516D7E6698E425A at byte 108467522 # Attribute pct total min max avg 95% stddev median # ========= ====== ======= ======= ======= ======= ======= ======= ======= # Count 2 317 # Exec time 2 944ms 0 205ms 3ms 2ms 22ms 108us # Hosts 181 10.5.9.20 (7/2%), 10.5.10.157 (6/1%)... 179 more # Databases 1 platform # Time range 2010-10-12 13:21:53.218711 to 2010-10-12 13:22:00.315719 # bytes 1 20.43k 66 68 66.01 65.89 0 65.89 # Errors 1 none # Rows affe 0 0 0 0 0 0 0 0 # Warning c 0 0 0 0 0 0 0 0 # Query_time distribution # 1us # 10us #################################################### # 100us ################################################################ # 1ms ######### # 10ms # # 100ms # # 1s # 10s+ # Tables # SHOW TABLE STATUS FROM `db` LIKE 'entry'G # SHOW CREATE TABLE `db`.`entry`G # EXPLAIN select id, type from entry where id = xxx G 2010 10 17
  22. 22. DB • # Query_time distribution # 1us # 10us #################################################### # 100us ###################################################### # 1ms ######### # 10ms # # 100ms # # 1s # 10s+ • • SQL explain 2010 10 17
  23. 23. • • • ↓ mk-query-digest tcpdump !! mysql dump OK 2010 10 17
  24. 24. DB • • • • • • 2010 10 17
  25. 25. DB • • $mysqladmin -uroot -p extended-STATUS | egrep '(Max|Threads_)' | Max_used_connections | 68 | | Threads_cached | 36 | | Threads_connected | 32 | | Threads_created | 68 | | Threads_running |5 | 2010 10 17
  26. 26. DB • • $ mysqladmin -uroot -p -hdbsrv status | awk '{print $22}' 9780.895 2010 10 17
  27. 27. DB • • $ sar 13:00:01 CPU %user %nice %system %iowait %idle 13:10:01 all 0.97 0.00 0.32 0.36 98.35 13:20:01 all 0.80 0.00 0.30 0.25 98.65 13:30:01 all 0.81 0.00 0.29 0.28 98.62 13:40:01 all 0.73 0.00 0.28 0.24 98.76 2010 10 17
  28. 28. ex. open Platform • • • • • 2010 10 17
  29. 29. • • from tcpdump & GadgetServer timeout • 100 200ms • ex. 2010 10 17
  30. 30. • • ping, traceroute etc • • 10ms ( ) • EC2 100ms 200ms 2010 10 17
  31. 31. • • platform <-> • <-> (Web <-> DB) • VM • I/O • 2010 10 17
  32. 32. • • ex. Platform @mbga OpenPlatform • TCO 2010 10 17
  33. 33. • • • DB • • • 2010 10 17
  34. 34. 2010 10 17
  35. 35. Thanks! 2010 10 17

×