"Ops Tools with Perl" 2012/05/12 Hokkaido.pm

OpsToolswith Perl Hokkaido.pm #7 2012/05/12 riywo

Self IntroductionRyosuke IWANAGAa.k.a. riywo(りーお)DeNA(2009∼) Mobage Server-side Ops-Engi / DBA / Manager

AcknowledgementThanks for JPA and Hokkaido.pm :) 地域PM向け講師派遣支援プログラム http://japan.perlassociation.org/ services/send-a-monger

But...Am I a Perl Monger?I’m not a Programer Mobage uses Perl heavily but I have never written the business logic...But, I read Perl heavily Because I should cope with all system trouble including application bugs ;PI wrote some operation tools with Perl And I also use some Perl tools

But...Am I a Perl Monger?So...I’ll try to talk about my knowledge, thought,and tools as much as possibleしょぼくてごめんなさい＞＜

Perl Ops-Tools Introduction

kazeburo-ware

http://blog.nomadscafe.jp/ @kazeburo

st ca re Fo d uloC http://www.slideshare.net/kazeburo/introduction-to-cloudforecast-yapcasia-2010-tokyo

CloudForecastServer Resource Collecting/MonitoringEasy install git clone & cpanm --installdeps . SNMP / RRDs is neededLightWeight - about 5,000 lines (*.pm)WAF: Shirahata.pmSee also リソースモニタリングツール「CloudForecast」入門 - As a Futurist... ( http://blog.riywo.com/2011/02/27/043646 )

s t ca re Fo th wroG http://blog.nomadscafe.jp/2011/12/growthforecast.html http://blog.nomadscafe.jp/2012/02/growthforecast1json.html http://d.hatena.ne.jp/tagomoris/20120218/1329558305

GrowthForecastAPI and Graph Render of RRDToolEasy install git clone & cpanm --installdeps . RRDs is neededLightWeight - about 1,800 lines (*.pm)WAF: Kossy.pmSee also GrowthForecastというグラフ表示ツールでる話 - blog.nomadscafe.jp ( http://blog.nomadscafe.jp/2011/12/growthforecast.html )

**Alien::RRDtool**RRDs is not CPAN module...Alien::RRDtool Install rrdtool as CPAN module extlib/lib/perl5/{archname}/RRDs.pm extlib/lib/perl5/auto/share/dist/Alien-RRDtool/Especially useful for perlbrew* Developed by gfxSee also Alien::RRDtool - RRDtoolをCPANから入れる - Perl Advent Calendar Japan 2011 Hacker Track( http://perl-users.jp/articles/advent- calendar/2011/hacker/2 )

chase-tail

chase-tail$ tail -f error_log | chase-tail -l 10 -t various_errorOne sheet perl script Highlight some keywords Detect fast flooding of logSee also @hirose31 垂れ流されるログのおしりを追いかける - (ひ)メモ( http:// d.hatena.ne.jp/hirose31/20120120/1327025989 )

App::Ikachan

App::Ikachan$ curl -F channel=#yappo -Fmessage=oppai http://hoge.com/noticeHTTP API of IRC posting use it from any languagesEasy install (git clone & cpanm) @YappoSee also YappoLogs: App::Ikachan - 様々なサーバのバッチ処理の結果等を IRC のチャンネルに通知するサーバ( http://blog.yappo.jp/yappo/ archives/000760.html )

Percona Toolkit

Percona ToolkitMixed tools - Maartkit and Aspersa Almost all tools are written in PerlSystem utility / MySQL Ops @Percona “pt-query-digest” is quite efficient for performance tuning of MySQL serverSee also Percona Toolkit - Percona Software( http://www.percona.com/ software/percona-toolkit/ )

Touryo

“Touryo” is conﬁguration management tool I needed a Configuration Management Tool Chef is good but difficult to customize So, I started to think a plan of Touryo Simple and Lightweight Testable able to write DSL in Perl freely

How to write DSL in Perl?I didn’t know it X(I asked gfx(@__gfx__) who is a great PerlMonger and works at DeNA

use strict; [dsl.pl]use warnings; test {Hoge->do_test; my $hoge = shift; print "Hello $hoge!n"; };#---------------------package Hoge;my $func;sub Hoge::test(&) { my $block = shift; $func = $block;}sub do_test { package Hoge; do "dsl.pl"; $func->("DSL");}

What can Touryo do? $ touryo test web01 web01 crontab apache FAIL? $ touryo run web01 sysctl fastcgi admin: touryo rpms appgame1:web - “touryo test host” - check the state of the host db01 db02 - correctly installed? crontab crontab - correctly conﬁgured? sysctl sysctl rpms rpms - “touryo run host” mysql - setup to the “correct” state mysql game1:db:master backup -> Then, what is the “correct” state? game1: db:backup

How to define the “correct” state? (1) Touryo config web01 role: game1 blueprint: crontab apache - crontab sysctl fastcgi - sysctl - rpms rpms app role: game1:web role: game1:dbgame1:web blueprint: blueprint: - apache - mysql db01 - fastcgi role: game1:db:backup db02 - app blueprint: crontab crontab - backup sysctl sysctl admin: rpms rpms touryo - Touryo use “blueprint” config mysql mysql - means what should be installed game1: - The config defined by “role” backupdb:master game1: - should be defined the role of hosts db:backup - inherited searching - separated by comma

What is the “blueprint”? crontab blueprint blueprint/crontab/root.tx web01 0 <: $cron_hour :> * * * /path/to/command crontab apache blueprint/crontab/blueprint.pl admin: touryo sysctl fastcgi bootstrap { my $c = shift; rpms app $c->add_attribute({ cron_hour => 5, game1:web cron => root.tx, });- blueprint locate the directory }; db01blueprint name - named the db02 crontab test {- contain “blueprint.pl”crontab my $c = shift; sysctl - Perl DSL for Touryo sysctl is_diff $c->sudo_ssh->crontab, - able to use templaterpms engine $c->template(cron)->content; rpms done_testing; - Text::Xslate::Syntax::Kolon }; mysql mysql - using “attribute” valuables game1: -db:master backup have other useful functions run { game1: my $c = shift; - sudo_ssh, template, crontab db:backup $c->sudo_ssh->crontab($c->template(cron)); - etc... };

What is the “attribute”? blueprint/crontab/root.tx web01 0 <: $cron_hour :> * * * /path/to/command admin: crontab apache blueprint/crontab/blueprint.pl touryo bootstrap { sysctl fastcgi my $c = shift; $c->add_attribute({ rpms app cron_hour => 5, game1:web cron => root.tx, }); };- “attribute” can be used to db01 db02change the behavior of test/run default rendering crontab crontab 0 5 * * * /path/to/command - changing logic, template, etc.. sysctl sysctl- overwriting of “attribute” If defined attribute on - bootstrap rpms rpms the Touryo Config ... - role config Touryo Config mysql mysql role: game1:web - defined for the role game1: backup attribute: -db:masterconfig node cron_hour: 9 game1: - defined for the node(host) db:backup 0 9 * * * /path/to/command

How to define the “correct” state? (2) Touryo config web01 role: game1 crontab apache blueprint: attribute: sysctl fastcgi - crontab somaxconn: 1024 - sysctl rpms app - rpmsgame1:web cron_hour: 9 somaxconn: 10240 role: game1:web fcgi_proc: 100 blueprint: attribute: - apache cron_hour: 9 - fastcgi somaxconn: 10240 web02 - app fcgi_proc: 100 crontab apache sysctl fastcgi node: web02 rpms app blueprint: attribute: admin: - git fcgi_proc: 10game1:web git touryo cron_hour: 9 - You can vary correct “blueprint” and somaxconn: 10240 “attribute” by Touryo config fcgi_proc: 10 - role config and node config

Lightweight/Testable/Freedomfind lib/ -name "*.pm" | xargs cat | wc -l about 1500 (including html templates)Like test scripts of Perl Software, you cantest the servers’ configurationyou can write anything on blueprint.pl Conversely you have to write anything

Other Featuresin terms of Programming use Amon2 easy to develop Web Interface and CLI use TAP::Parser to get the test result test function uses Test::More use IPC::Cmd and “ssh” / “sudo ssh” easy to configure with .ssh/config or others blueprint dependance resolving easy to cascade the config

What languageshould I use?

There are many languagesscript / compile Perl,Ruby,Python,PHP / C,C++,Javaprocedural / OO / functional Perl,C / Java,Ruby / Scala,Haskellpopular / minority ???(religious problem...)

My ThoughtYou can use any languages you like Engineer have to treat multi languagesDevelopment speed / Maintenance cost They are usually trade-off Popular LLs are often suited for both Simple / Light / Thin / Useful Modules Everyone can understand it easily Perl/Ruby/Python are similar

I ♡ PerlFastest to develop for me I am used to Perl because DeNA use Perl There are many Perl Mongers around :) colleague and friendsCPAN has many useful modules long history and super hackers

Finally...

DeNA ♡ EngineersWe need more powerful engineersWe use Perl and other languages Ruby/JavaScript/C/C++/Java/Objective-C SocialGame/Platform/SDK/Data miningOr let’s make very useful tools with me :) Server Ops/Database Admin/NetworkIf you are interested in us, pleeeeeeeeeasecontact me or DeNA staffs! hirose31, gfx, hidek, zigorou, nekokak...

Thanks! http://blog.riywo.com http://twitter.com/riywoThanks for JPA, @kazeburo, @hirose31, @Yappo, @Percona, and All Perl Mongers!

http://hokkaido.pm.org	858
http://blog.riywo.com	308
http://www.perl-hokkaido.org	3
http://a0.twimg.com	2
http://webcache.googleusercontent.com	2
https://si0.twimg.com	1

"Ops Tools with Perl" 2012/05/12 Hokkaido.pm

by riywo on May 12, 2012

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 1,174

Statistics

"Ops Tools with Perl" 2012/05/12 Hokkaido.pm Presentation Transcript