Web Operations and Perl kansai.pm#14

5,239 views
5,144 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,239
On SlideShare
0
From Embeds
0
Number of Embeds
1,957
Actions
Shares
0
Downloads
12
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Web Operations and Perl kansai.pm#14

    1. 1. Web Operations & PerlInside livedoor Blog s Backend and cloudforecast Kansai.pm #14 Masahiro Nagano (kazeburo)
    2. 2. 謝辞Kansai.pm++AzureStone++ はてな++ JPA++
    3. 3. Me• 長野雅広• @kazeburo• PAUSE: KAZEBURO• 横浜在住• Operations Engineer, livedoor • NHN Japan (2012• http://blog.nomadscafe.jp/
    4. 4. livedoor
    5. 5. Our services2011年9月26日月曜日
    6. 6. Large scale web service 2800+ servers 3200+ hosts
    7. 7. large traffic4GbpsAug 2009
    8. 8. large traffic16Gbps Nov 2011 自社回線 + CDN
    9. 9. livedoor Blog 2003-12∼
    10. 10. Stats•blog開設数 400万•月間UU 3000万 (2011年6月/コムスコア調べ)•月間1億PVを超えるブログ •痛いニュース •ハムスター速報
    11. 11. Infrastructure PC mobile PC mobileCMS APP CMS APP APP APP STF Analyzer Job Main Clusteredmemcached Queue DB DB ブログ 画像 解析
    12. 12. Softwares•Apache + modules•mod_perl / Plack PSGI•MySQL 4.0.x 5.1.x•TheSchwartz / Q4M•memcached
    13. 13. STF 画像ストレージ MogileFS ライクな 分散オブジェクトストレージ3億オブジェクト/10億以上のファイル 20億ファイル以上(移行中) 近日オープンソース化予定!
    14. 14. STF Internal nginx • dispatcher Squid ➡ 独自Apache module • jobqueue MetaDispatcher JobQueue ➡ STOMP DB ➡ perl • Meta DBmod_dav mod_dav mod_dav mod_dav ➡ MySQLmod_dav mod_dav mod_dav mod_dav
    15. 15. STF Internal nginx • dispatcher Squid ➡ Plack/PSGI ➡ Apache mod_reproxy MetaDispatcher JobQueue DB • jobqueue ➡ Q4M ➡ perlmod_dav mod_dav mod_dav mod_dav • Meta DB ➡ XtraDBmod_dav mod_dav mod_dav mod_dav
    16. 16. Traffic Control 複数の回線 自社回線2 コスト削減 自社回線1 CDN 一定量の帯域維持 インフラの制限 nginx nginx nginxnginxのrewriteでは表現難しい Squid Squid Squid
    17. 17. Traffic Control 自社回線2 自社回線1 CDN nginx nginx nginx Squid Squid Squid
    18. 18. Traffic Control$ ./configure --with-http_perl_module# nginx.confperl_set $usecdn 自社回線2sub { my $r = shift; my @lt = localtime(); return "AB" 自社回線1 CDN if $lt[2] >= 20 || $lt[2] < 3; return "B" if $lt[2] >= 18 || $lt[2] < 3; return "";}; nginx nginx nginxif ( $usecdn ~ A ) { rewrite /a http://cdn/$1 redirect;} Squid Squid Squidif ( $usecdn ~ B ) { rewrite /b http://cdn/$1 redirect;}
    19. 19. Traffic Control
    20. 20. Traffic Control
    21. 21. MySQL MySQL 4.0.x Software based shardingOver 160 clusters / 80 servers(Master-Slave*2) 2.0TB of ibdata (master only) App ① ② Mapping Cluster Cluster Cluster Cluster Cluster (Main DB) 1 2 3 4 5
    22. 22. Backup of MySQL 4.0 $ mysqldump --master-data blocks update query MySQL 5.1.x MySQL 4.0.xFLUSH LOCAL TABLES FLUSH TABLES WITH READ LOCKFLUSH TABLES WITH READ LOCK BEGINSET SESSION TRANSACTION DB選択ISOLATION LEVEL REPEATABLE READ テーブルからデータの読み込みSTART TRANSACTION WITH COMMIT;CONSISTENT SNAPSHOT SHOW MASTER STATUSSHOW MASTER STATUS UNLOCK TABLES;UNLOCK TABLESDB選択テーブルからデータの読み込み
    23. 23. mysqldump wrapper for MySQL 4.0$dbh->do(FLUSH TABLES WITH READ LOCK);$dbh->select(SHOW MASTER STATUS);pipe my $logrh, my $logwh;my $pid = forkif ( $pid == 0 ) { #子プロセス open STDOUT, >&, $logwh; exec("mysqldump","--single-transaction","--databases","...");}while(<$logrh>){ print; if ( m!^CREATE DATABASE!) { $dbh->do(UNLOCK TABLE); }} https://github.com/kazeburo/mysql40dump
    24. 24. OperationsData Hotel (監視・一時切り分け) Nagios cloudforecast
    25. 25. easy Nagiosconfiguration generatoruse Text::Xslate;use Data::Section::Simple qw(get_data_section);for my $l ( split /n/, get_data_section(data) ) { chomp($l); my ($ip, $host) = split /s+/, $l; push @nodes, { ipaddr => $ip, hostname => $host, };}my $tx = Text::Xslate->new();print $tx->render_string(get_data_section(tmpl), { nodes => @nodes });__DATA__@@ data10.x.x.201 blogserver110.x.x.202 blogserver2@@ tmpl...
    26. 26. @@ tmpl: for $nodes -> $node {define host{ use generic-host max_check_attempts 3 host_name <: $node.hostname :> alias <: $node.hostname :> address <: $node.ipaddr :>}define service{ use generic-service host_name <: $node.hostname :> service_description Disk Usage notification_options u,c,r check_command check_disk}define service{ use generic-service host_name <: $node.hostname :> service_description MySQL Uptime max_check_attempts 1 notification_options u,c check_command check_mysql_uptime}:}
    27. 27. What monitoring with Nagios Total Traffic nginx stub_status MySQL uptime MySQL replication delay
    28. 28. cloudforecastMetrics収集(リソースモニタリングについて) http://www.flickr.com/photos/gsfc/4427417620/in/set-72157623639063326/
    29. 29. グラフ大好き!
    30. 30. 監視とMetrics収集 Server Server agent agent fetchpolling 監視サーバ 監視サーバ update DB/Graph notify browse 運用者 運用者 監視 Metrics収集
    31. 31. Why needs Metrics パフォーマンス障害を未然に防ぐ アプリケーションの改善サイクル
    32. 32. cloudforecast Metrics gathering tool(リソースモニタリングツール) Open Source, Perl RRDTool wrapper Easy deployment Fast and Extensible
    33. 33. cloudforecast core pluginsBasic (CPU usage, Load Average,Memory usage, TCP Established) Traffic, Disk Usage, Disk IO Apache, Nginx, Squid MySQL, InnoDB, memcached
    34. 34. cloudforecast users livedoorと数社
    35. 35. cloudforecast at livedoor 2 instances 1000 hosts/instance(max) wrote some plugins (TheSchwartz, Q4M, qmail, postfix, LoadBalancer) integrate with Yabitz* *host mangement application (https://github.com/tagomoris/yabitz)
    36. 36. cloudforecastarchitecture#1 Web RRD DB radar pull via snmp,http,DBI.. Server
    37. 37. cloudforecast architecture#2 RRD DB updater updaterradar enqueue gearmand fetcher fetcher fetcher fetcher fetcher Server Server Server Server Server
    38. 38. How to setup & configurationhttp://www.slideshare.net/kazeburo/introduction-to-cloudforecast-yapcasia-2010-tokyo
    39. 39. Web Interface#1
    40. 40. Web Interface#2
    41. 41. Web Interface#3
    42. 42. Web Interface#4
    43. 43. Web Interface#5
    44. 44. cloudforecasttuening guide
    45. 45. CPU usage#1 (App) user > アプリケーションの見直し > サーバ増設
    46. 46. CPU usage#2 (DB) iowait > クエリ・インデックスの確認 > メモリ増設・スケールアップ
    47. 47. Memory usage (App) OOM! > ApacheのMaxClientsを下げてOOM回避 > ループの可能性があるのでコード調査
    48. 48. MySQL slow-log> query-digest を使用し、今のうちから調査を
    49. 49. InnoDB disabled..> 意図的に無効にされているのでなければ再構築..
    50. 50. MySQL Query vs Row read750r/q > indexが効率的に使えていない可能性 > update/deleteクエリにも要注意
    51. 51. Nginx & network> writing の上昇 = networkが詰まっている可能性
    52. 52. まとめ
    53. 53. 「推測するな、計測せよ」 #!/usr/bin/perl use Metrics; pdca() while 1;
    54. 54. ご清聴ありがとう ございました

    ×