Your SlideShare is downloading. ×
0
OpsToolswith Perl Hokkaido.pm #7   2012/05/12      riywo
Self IntroductionRyosuke IWANAGAa.k.a. riywo(りーお)DeNA(2009∼) Mobage Server-side Ops-Engi / DBA / Manager
AcknowledgementThanks for JPA and Hokkaido.pm :) 地域PM向け講師派遣支援プログラム   http://japan.perlassociation.org/   services/send-a-m...
But...Am I a Perl Monger?I’m not a Programer  Mobage uses Perl heavily but I have  never written the business logic...But,...
But...Am I a Perl Monger?So...I’ll try to talk about my knowledge, thought,and tools as much as possibleしょぼくてごめんなさい><
Perl Ops-Tools Introduction
kazeburo-ware
http://blog.nomadscafe.jp/        @kazeburo
st              ca            re     Fo    d    uloC        http://www.slideshare.net/kazeburo/introduction-to-cloudforeca...
CloudForecastServer Resource Collecting/MonitoringEasy install  git clone & cpanm --installdeps .  SNMP / RRDs is neededLi...
s   t               ca            re        Fo     th    wroG        http://blog.nomadscafe.jp/2011/12/growthforecast.html...
GrowthForecastAPI and Graph Render of RRDToolEasy install  git clone & cpanm --installdeps .  RRDs is neededLightWeight - ...
**Alien::RRDtool**RRDs is not CPAN module...Alien::RRDtool Install rrdtool as CPAN module  extlib/lib/perl5/{archname}/RRD...
chase-tail
chase-tail$ tail -f error_log | chase-tail -l 10 -t various_errorOne sheet perl script  Highlight some keywords  Detect fa...
App::Ikachan
App::Ikachan$ curl -F channel=#yappo -Fmessage=oppai http://hoge.com/noticeHTTP API of IRC posting  use it from any langua...
Percona Toolkit
Percona ToolkitMixed tools - Maartkit and Aspersa Almost all tools are written in PerlSystem utility / MySQL Ops          ...
Touryo
“Touryo” is configuration   management tool I needed a Configuration Management Tool   Chef is good but difficult to custom...
How to write DSL in       Perl?I didn’t know it X(I asked gfx(@__gfx__) who is a great PerlMonger and works at DeNA
use strict;     [dsl.pl]use warnings;                 test {Hoge->do_test;       my $hoge = shift;                     pri...
What can Touryo do?                                 $ touryo test web01            web01      crontab        apache       ...
How to define the “correct” state? (1)                                  Touryo config            web01                  role...
What is the “blueprint”?                                       crontab blueprint                                      blue...
What is the “attribute”?                                      blueprint/crontab/root.tx             web01                 ...
How to define the “correct” state? (2)                                  Touryo config            web01                      ...
Lightweight/Testable/Freedomfind lib/ -name "*.pm" | xargs cat | wc -l  about 1500 (including html templates)Like test scr...
Other Featuresin terms of Programming use Amon2   easy to develop Web Interface and CLI use TAP::Parser to get the test re...
What languageshould I use?
There are many    languagesscript / compile  Perl,Ruby,Python,PHP / C,C++,Javaprocedural / OO / functional  Perl,C / Java,...
My ThoughtYou can use any languages you like  Engineer have to treat multi languagesDevelopment speed / Maintenance cost  ...
I ♡ PerlFastest to develop for me I am used to Perl because DeNA use Perl There are many Perl Mongers around :)   colleagu...
Finally...
DeNA ♡ EngineersWe need more powerful engineersWe use Perl and other languages  Ruby/JavaScript/C/C++/Java/Objective-C  So...
Thanks!           http://blog.riywo.com          http://twitter.com/riywoThanks for JPA, @kazeburo, @hirose31, @Yappo, @Pe...
"Ops Tools with Perl" 2012/05/12 Hokkaido.pm
Upcoming SlideShare
Loading in...5
×

"Ops Tools with Perl" 2012/05/12 Hokkaido.pm

3,048

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,048
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
8
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of ""Ops Tools with Perl" 2012/05/12 Hokkaido.pm"

  1. 1. OpsToolswith Perl Hokkaido.pm #7 2012/05/12 riywo
  2. 2. Self IntroductionRyosuke IWANAGAa.k.a. riywo(りーお)DeNA(2009∼) Mobage Server-side Ops-Engi / DBA / Manager
  3. 3. AcknowledgementThanks for JPA and Hokkaido.pm :) 地域PM向け講師派遣支援プログラム http://japan.perlassociation.org/ services/send-a-monger
  4. 4. But...Am I a Perl Monger?I’m not a Programer Mobage uses Perl heavily but I have never written the business logic...But, I read Perl heavily Because I should cope with all system trouble including application bugs ;PI wrote some operation tools with Perl And I also use some Perl tools
  5. 5. But...Am I a Perl Monger?So...I’ll try to talk about my knowledge, thought,and tools as much as possibleしょぼくてごめんなさい><
  6. 6. Perl Ops-Tools Introduction
  7. 7. kazeburo-ware
  8. 8. http://blog.nomadscafe.jp/ @kazeburo
  9. 9. st ca re Fo d uloC http://www.slideshare.net/kazeburo/introduction-to-cloudforecast-yapcasia-2010-tokyo
  10. 10. CloudForecastServer Resource Collecting/MonitoringEasy install git clone & cpanm --installdeps . SNMP / RRDs is neededLightWeight - about 5,000 lines (*.pm)WAF: Shirahata.pmSee also リソースモニタリングツール「CloudForecast」入門 - As a Futurist... ( http://blog.riywo.com/2011/02/27/043646 )
  11. 11. s t ca re Fo th wroG http://blog.nomadscafe.jp/2011/12/growthforecast.html http://blog.nomadscafe.jp/2012/02/growthforecast1json.html http://d.hatena.ne.jp/tagomoris/20120218/1329558305
  12. 12. GrowthForecastAPI and Graph Render of RRDToolEasy install git clone & cpanm --installdeps . RRDs is neededLightWeight - about 1,800 lines (*.pm)WAF: Kossy.pmSee also GrowthForecastというグラフ表示ツールで る話 - blog.nomadscafe.jp ( http://blog.nomadscafe.jp/2011/12/growthforecast.html )
  13. 13. **Alien::RRDtool**RRDs is not CPAN module...Alien::RRDtool Install rrdtool as CPAN module extlib/lib/perl5/{archname}/RRDs.pm extlib/lib/perl5/auto/share/dist/Alien-RRDtool/Especially useful for perlbrew* Developed by gfxSee also Alien::RRDtool - RRDtoolをCPANから入れる - Perl Advent Calendar Japan 2011 Hacker Track( http://perl-users.jp/articles/advent- calendar/2011/hacker/2 )
  14. 14. chase-tail
  15. 15. chase-tail$ tail -f error_log | chase-tail -l 10 -t various_errorOne sheet perl script Highlight some keywords Detect fast flooding of logSee also @hirose31 垂れ流されるログのおしりを追いかける - (ひ)メモ( http:// d.hatena.ne.jp/hirose31/20120120/1327025989 )
  16. 16. App::Ikachan
  17. 17. App::Ikachan$ curl -F channel=#yappo -Fmessage=oppai http://hoge.com/noticeHTTP API of IRC posting use it from any languagesEasy install (git clone & cpanm) @YappoSee also YappoLogs: App::Ikachan - 様々なサーバのバッチ処理の結果等を IRC のチャンネルに通知するサーバ( http://blog.yappo.jp/yappo/ archives/000760.html )
  18. 18. Percona Toolkit
  19. 19. Percona ToolkitMixed tools - Maartkit and Aspersa Almost all tools are written in PerlSystem utility / MySQL Ops @Percona “pt-query-digest” is quite efficient for performance tuning of MySQL serverSee also Percona Toolkit - Percona Software( http://www.percona.com/ software/percona-toolkit/ )
  20. 20. Touryo
  21. 21. “Touryo” is configuration management tool I needed a Configuration Management Tool Chef is good but difficult to customize So, I started to think a plan of Touryo Simple and Lightweight Testable able to write DSL in Perl freely
  22. 22. How to write DSL in Perl?I didn’t know it X(I asked gfx(@__gfx__) who is a great PerlMonger and works at DeNA
  23. 23. use strict; [dsl.pl]use warnings; test {Hoge->do_test; my $hoge = shift; print "Hello $hoge!n"; };#---------------------package Hoge;my $func;sub Hoge::test(&) { my $block = shift; $func = $block;}sub do_test { package Hoge; do "dsl.pl"; $func->("DSL");}
  24. 24. What can Touryo do? $ touryo test web01 web01 crontab apache FAIL? $ touryo run web01 sysctl fastcgi admin: touryo rpms appgame1:web - “touryo test host” - check the state of the host db01 db02 - correctly installed? crontab crontab - correctly configured? sysctl sysctl rpms rpms - “touryo run host” mysql - setup to the “correct” state mysql game1:db:master backup -> Then, what is the “correct” state? game1: db:backup
  25. 25. How to define the “correct” state? (1) Touryo config web01 role: game1 blueprint: crontab apache - crontab sysctl fastcgi - sysctl - rpms rpms app role: game1:web role: game1:dbgame1:web blueprint: blueprint: - apache - mysql db01 - fastcgi role: game1:db:backup db02 - app blueprint: crontab crontab - backup sysctl sysctl admin: rpms rpms touryo - Touryo use “blueprint” config mysql mysql - means what should be installed game1: - The config defined by “role” backupdb:master game1: - should be defined the role of hosts db:backup - inherited searching - separated by comma
  26. 26. What is the “blueprint”? crontab blueprint blueprint/crontab/root.tx web01 0 <: $cron_hour :> * * * /path/to/command crontab apache blueprint/crontab/blueprint.pl admin: touryo sysctl fastcgi bootstrap { my $c = shift; rpms app $c->add_attribute({ cron_hour => 5, game1:web cron => root.tx, });- blueprint locate the directory }; db01blueprint name - named the db02 crontab test {- contain “blueprint.pl”crontab my $c = shift; sysctl - Perl DSL for Touryo sysctl is_diff $c->sudo_ssh->crontab, - able to use templaterpms engine $c->template(cron)->content; rpms done_testing; - Text::Xslate::Syntax::Kolon }; mysql mysql - using “attribute” valuables game1: -db:master backup have other useful functions run { game1: my $c = shift; - sudo_ssh, template, crontab db:backup $c->sudo_ssh->crontab($c->template(cron)); - etc... };
  27. 27. What is the “attribute”? blueprint/crontab/root.tx web01 0 <: $cron_hour :> * * * /path/to/command admin: crontab apache blueprint/crontab/blueprint.pl touryo bootstrap { sysctl fastcgi my $c = shift; $c->add_attribute({ rpms app cron_hour => 5, game1:web cron => root.tx, }); };- “attribute” can be used to db01 db02change the behavior of test/run default rendering crontab crontab 0 5 * * * /path/to/command - changing logic, template, etc.. sysctl sysctl- overwriting of “attribute” If defined attribute on - bootstrap rpms rpms the Touryo Config ... - role config Touryo Config mysql mysql role: game1:web - defined for the role game1: backup attribute: -db:masterconfig node cron_hour: 9 game1: - defined for the node(host) db:backup 0 9 * * * /path/to/command
  28. 28. How to define the “correct” state? (2) Touryo config web01 role: game1 crontab apache blueprint: attribute: sysctl fastcgi - crontab somaxconn: 1024 - sysctl rpms app - rpmsgame1:web cron_hour: 9 somaxconn: 10240 role: game1:web fcgi_proc: 100 blueprint: attribute: - apache cron_hour: 9 - fastcgi somaxconn: 10240 web02 - app fcgi_proc: 100 crontab apache sysctl fastcgi node: web02 rpms app blueprint: attribute: admin: - git fcgi_proc: 10game1:web git touryo cron_hour: 9 - You can vary correct “blueprint” and somaxconn: 10240 “attribute” by Touryo config fcgi_proc: 10 - role config and node config
  29. 29. Lightweight/Testable/Freedomfind lib/ -name "*.pm" | xargs cat | wc -l about 1500 (including html templates)Like test scripts of Perl Software, you cantest the servers’ configurationyou can write anything on blueprint.pl Conversely you have to write anything
  30. 30. Other Featuresin terms of Programming use Amon2 easy to develop Web Interface and CLI use TAP::Parser to get the test result test function uses Test::More use IPC::Cmd and “ssh” / “sudo ssh” easy to configure with .ssh/config or others blueprint dependance resolving easy to cascade the config
  31. 31. What languageshould I use?
  32. 32. There are many languagesscript / compile Perl,Ruby,Python,PHP / C,C++,Javaprocedural / OO / functional Perl,C / Java,Ruby / Scala,Haskellpopular / minority ???(religious problem...)
  33. 33. My ThoughtYou can use any languages you like Engineer have to treat multi languagesDevelopment speed / Maintenance cost They are usually trade-off Popular LLs are often suited for both Simple / Light / Thin / Useful Modules Everyone can understand it easily Perl/Ruby/Python are similar
  34. 34. I ♡ PerlFastest to develop for me I am used to Perl because DeNA use Perl There are many Perl Mongers around :) colleague and friendsCPAN has many useful modules long history and super hackers
  35. 35. Finally...
  36. 36. DeNA ♡ EngineersWe need more powerful engineersWe use Perl and other languages Ruby/JavaScript/C/C++/Java/Objective-C SocialGame/Platform/SDK/Data miningOr let’s make very useful tools with me :) Server Ops/Database Admin/NetworkIf you are interested in us, pleeeeeeeeeasecontact me or DeNA staffs! hirose31, gfx, hidek, zigorou, nekokak...
  37. 37. Thanks! http://blog.riywo.com http://twitter.com/riywoThanks for JPA, @kazeburo, @hirose31, @Yappo, @Percona, and All Perl Mongers!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×