0
Assurer – A server testing/monitoring framework Gosuke Miyashita http://mizzy.org/
Myself <ul><li>Known as Chocoboy Miyashita somewhere </li></ul><ul><li>Working in paperboy&co. </li></ul><ul><li>Mainly de...
 
What’s Assurer ? <ul><li>A server testing/monitoring framework like Plagger  </li></ul><ul><ul><li>Plugin mechanism </li><...
Where the name Assurer derived from ? <ul><li>Pronounce “A-shu-ra” </li></ul><ul><li>Plagger like tools ended with “er” </...
Image of exec Assurer #0
Image of exec Assurer #1
Image of Asura Baster
Image of the real Asura
Image of fake Asura
Execution phases of Assurer
Execution phases of Assurer  <ul><li>Test phase </li></ul><ul><ul><li>Execute test </li></ul></ul><ul><li>Nofity phase </l...
Relation of each phases Test Publish Format Nofity
Details of each phases
Test phase
Test phase <ul><li>Execute test </li></ul><ul><li>Plugins on this phase: </li></ul><ul><ul><li>Test::HTTP </li></ul></ul><...
Config of test phase <ul><li>test: </li></ul><ul><li>-  module:  HTTP </li></ul><ul><li>config: </li></ul><ul><li>host:   ...
Notify phase
Notify  フェーズ <ul><li>Notify test results </li></ul><ul><li>Plugin on this phase: </li></ul><ul><ul><li>Notify::IRC </li></...
Config of notify phase <ul><li>notify: </li></ul><ul><li>-  module:  IRC </li></ul><ul><li>config: </li></ul><ul><li>daemo...
Notify::IRC example
Format phase
Format phase <ul><li>Format test results </li></ul><ul><li>Plugins on this phase: </li></ul><ul><ul><li>Format::Text </li>...
Config of format phase <ul><li>format: </li></ul><ul><li>-  module:  HTML </li></ul><ul><li>config: </li></ul><ul><li>css:...
Publish phase
Publish phase <ul><li>Output formatted results </li></ul><ul><li>Plugins on this phase: </li></ul><ul><ul><li>Publish::Ter...
Config of publish phase <ul><li>publish: </li></ul><ul><li>- module: Mail </li></ul><ul><li>config: </li></ul><ul><li>subj...
Format::Text + Publish::Term
Format::HTML + Publish::File
Format::Text + Publish::Mail
Summary for now <ul><li>Assurer is a serverPlagger  </li></ul><ul><li>Test, Notify, Format and Publish phases </li></ul><u...
Extra functions
Hosts and Roles
Hosts and roles <ul><li>Write target hosts and roles in one place on config.yaml </li></ul>
Config without hosts and roles test: - module: HTTP config: host: www0.mizzy.org - module: HTTP config: host: www1.mizzy.o...
Config with hosts and roles test: - module: HTTP role: web hosts: web: - www0.mizzy.org - www1.mizzy.org
Filter
Filter <ul><li>Filtering test results on format and notify phase </li></ul><ul><li>Filtering formatted results on publish ...
Filter on format phase <ul><li>Only OK results are formatted as text format </li></ul>format: - module: Text filter: modul...
Filter on notify phase <ul><li>Only NOK results are notified to IRC </li></ul>notify: - module: IRC filter: module:  Statu...
Filter on publish phase <ul><li>Text formatted results are displayed on a terminal,html formatted results are sent as a ma...
Config validation
Config validation <ul><li>Kwalify – YAML schema validator  http://www.kuwata-lab.com/kwalify /   </li></ul><ul><li>Validat...
Mistake on common config test s :  # test is right - module: Test::HTTP  $ ./assurer.pl -c config.yaml Assurer::ConfigLoad...
Mistake on plugin config test: - module: HTTP config: content s : XXX  # content is right $ ./assurer.pl -c config.yaml As...
Shell mode
Shell mode <ul><li>Execute shell command on hosts in config.yaml </li></ul>
Shell mode $ ./assurer.pl --shell assurer>   uptime [www.mizzy.org]  21:48:24 up 5 days, 23:08, 2 users, load average: 0.2...
Exec on hosts belong to specified role $ ./assurer.pl --shell  --role=web assurer>   uptime [www.mizzy.org]  21:48:24 up 5...
Special command !on assurer>   !on   app1.foo.com     app2.foo.com   do  uptime # Exec on app1.foo.com and app2.foo.com on...
Special command !with assurer>   !with   web db   do  uptime # exec on hosts belong to web role and db role assurer>   !wi...
Special command !test assurer>   !test  HTTP Assurer::Plugin::Test::HTTP [info] Testing HTTP on www0.mizzy.org ...  ok 1 -...
Special command !test assurer>   !test  HTTP  on  app1.foo.com  # test http on app1.foo.com assurer>   !test  HTTP  on  /....
Parallel testing
Pallarel testing <ul><li>Multiple test processes on test phase </li></ul><ul><li>Default is 5 </li></ul><ul><li>Can change...
Distributed testing
Distributed testing localhost test0.mizzy.org www0.mizzy.org www1.mizzy.org www2.mizzy.org test1.mizzy.org test2.mizzy.org...
Config for distributed testing exec_on: -  host:  test0.mizzy.org priority:  3 -  host:  test1.mizzy.org priority:  2  -  ...
Auto config generation
Auto config generation <ul><li>Scan hosts in template.yaml by Nmap::Scanner </li></ul><ul><li>Detect services working on t...
Integragte Test::WWW::Mechanize
Integrate Test::WWW::Mechanize <ul><li>Can write tests using Test::WWW::Mechanize  </li></ul><ul><li>Put .pl files under a...
asset file for Test::WWW::Mechanize # $host and $mech are set by Assure core $mech ->get_ok(&quot;http:// $host &quot;, &q...
Inside Assurer
Assurer::Test module
Assurer::Test module <ul><li>Export Test::More like methods </li></ul><ul><li>is,() like(), ok() </li></ul><ul><li>Output ...
Inside test plugins
Inside test plugins package Assurer::Plugin::Test::HTTP; use base qw( Assurer::Plugin::Test ); use Assurer::Test; sub regi...
Internal mechanism in test phase
Internal mechanism in test phase assurer.pl Assurer::bootstrap() Assurer::Dispatch::run() call exec directly or  by ssh ex...
Assurer::Dispatch::run() <ul><li>Exec assurer_test.pl for each test  </li></ul><ul><li>Pallarelize by POE::Wheel::Run </li...
assurer_test.pl assurer_test.pl   -– config = LS0tCmNvbmZpZzoKICBjb250ZW50OiBJ dCB3b3JrcyEKICBob3N0OiB3d3cubWl6enkub3JnCm1...
Why POE::Wheel::Run ? <ul><li>Would like to exec tests as other processes </li></ul><ul><ul><li>Need non-blocking I/O with...
Roadmap
Roadmap <ul><li>Integrate with Nagios/NRPE (Test::Nagios) </li></ul><ul><li>Enhance notify phase </li></ul><ul><li>Store P...
Web site and IRC channels
Web sites and IRC channels <ul><li>http://assurer.jp/trac </li></ul><ul><li>http://mizzy.org/ </li></ul><ul><li>#assurer@c...
Thanks
Upcoming SlideShare
Loading in...5
×

Assurer - a pluggable server testing/monitoring framework

1,418

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,418
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Assurer - a pluggable server testing/monitoring framework"

  1. 1. Assurer – A server testing/monitoring framework Gosuke Miyashita http://mizzy.org/
  2. 2. Myself <ul><li>Known as Chocoboy Miyashita somewhere </li></ul><ul><li>Working in paperboy&co. </li></ul><ul><li>Mainly developed public/private web service APIs with perl and catalyt </li></ul><ul><li>... but now also woking on servers and networks and so on </li></ul><ul><li>Today’s my son’s entrance ceremony of an elementary school </li></ul>
  3. 4. What’s Assurer ? <ul><li>A server testing/monitoring framework like Plagger </li></ul><ul><ul><li>Plugin mechanism </li></ul></ul><ul><ul><li>YAML configurable </li></ul></ul><ul><ul><li>assets </li></ul></ul><ul><ul><li>Test by Test::Base </li></ul></ul><ul><li>Originally developed for one time test of servers that are newly built </li></ul><ul><li>Monitoring seems like testing, so decided to cover monitoring </li></ul>
  4. 5. Where the name Assurer derived from ? <ul><li>Pronounce “A-shu-ra” </li></ul><ul><li>Plagger like tools ended with “er” </li></ul><ul><ul><li>Archer by tokuhirom </li></ul></ul><ul><ul><li>Observer by Hatena (not in public) </li></ul></ul><ul><ul><li>Dishuber by Yappo </li></ul></ul><ul><ul><li>Precure by Kan Fushihara </li></ul></ul><ul><li>Testing is also called “Quality Assurance” </li></ul>MISS
  5. 6. Image of exec Assurer #0
  6. 7. Image of exec Assurer #1
  7. 8. Image of Asura Baster
  8. 9. Image of the real Asura
  9. 10. Image of fake Asura
  10. 11. Execution phases of Assurer
  11. 12. Execution phases of Assurer <ul><li>Test phase </li></ul><ul><ul><li>Execute test </li></ul></ul><ul><li>Nofity phase </li></ul><ul><ul><li>Notify test results </li></ul></ul><ul><li>Format phase </li></ul><ul><ul><li>Format test results </li></ul></ul><ul><li>Publish phase </li></ul><ul><ul><li>Output formatted test results </li></ul></ul>
  12. 13. Relation of each phases Test Publish Format Nofity
  13. 14. Details of each phases
  14. 15. Test phase
  15. 16. Test phase <ul><li>Execute test </li></ul><ul><li>Plugins on this phase: </li></ul><ul><ul><li>Test::HTTP </li></ul></ul><ul><ul><li>Test::SMTP </li></ul></ul><ul><ul><li>Test::DBI </li></ul></ul><ul><ul><li>Test::Ping </li></ul></ul><ul><ul><li>etc </li></ul></ul>
  16. 17. Config of test phase <ul><li>test: </li></ul><ul><li>- module: HTTP </li></ul><ul><li>config: </li></ul><ul><li>host: www.mizzy.org </li></ul><ul><li>content: It works! </li></ul>
  17. 18. Notify phase
  18. 19. Notify フェーズ <ul><li>Notify test results </li></ul><ul><li>Plugin on this phase: </li></ul><ul><ul><li>Notify::IRC </li></ul></ul>
  19. 20. Config of notify phase <ul><li>notify: </li></ul><ul><li>- module: IRC </li></ul><ul><li>config: </li></ul><ul><li>daemon_port: 9991 </li></ul><ul><li>nickname: assurerbot </li></ul><ul><li>server_host: chat.freenode.net </li></ul><ul><li>server_port: 6667 </li></ul><ul><li>server_channels: </li></ul><ul><li>- #assurer </li></ul><ul><li>- #assurer-jp </li></ul><ul><li>charset: iso-2022-jp </li></ul><ul><li>announce: notice </li></ul>
  20. 21. Notify::IRC example
  21. 22. Format phase
  22. 23. Format phase <ul><li>Format test results </li></ul><ul><li>Plugins on this phase: </li></ul><ul><ul><li>Format::Text </li></ul></ul><ul><ul><li>Fomat::HTML </li></ul></ul>
  23. 24. Config of format phase <ul><li>format: </li></ul><ul><li>- module: HTML </li></ul><ul><li>config: </li></ul><ul><li>css: /css/style.css </li></ul>
  24. 25. Publish phase
  25. 26. Publish phase <ul><li>Output formatted results </li></ul><ul><li>Plugins on this phase: </li></ul><ul><ul><li>Publish::Term </li></ul></ul><ul><ul><li>Publish::Mail </li></ul></ul><ul><ul><li>Publish::File </li></ul></ul>
  26. 27. Config of publish phase <ul><li>publish: </li></ul><ul><li>- module: Mail </li></ul><ul><li>config: </li></ul><ul><li>subject: Test results from Assurer </li></ul><ul><li>to: someone@example.com </li></ul><ul><li>from: root@example.com </li></ul>publish: - module: Mail config: subject: Test results from Assurer to: someone@example.com from: root@example.com
  27. 28. Format::Text + Publish::Term
  28. 29. Format::HTML + Publish::File
  29. 30. Format::Text + Publish::Mail
  30. 31. Summary for now <ul><li>Assurer is a serverPlagger </li></ul><ul><li>Test, Notify, Format and Publish phases </li></ul><ul><li>Easily change tests and outputs with plugins </li></ul>
  31. 32. Extra functions
  32. 33. Hosts and Roles
  33. 34. Hosts and roles <ul><li>Write target hosts and roles in one place on config.yaml </li></ul>
  34. 35. Config without hosts and roles test: - module: HTTP config: host: www0.mizzy.org - module: HTTP config: host: www1.mizzy.org
  35. 36. Config with hosts and roles test: - module: HTTP role: web hosts: web: - www0.mizzy.org - www1.mizzy.org
  36. 37. Filter
  37. 38. Filter <ul><li>Filtering test results on format and notify phase </li></ul><ul><li>Filtering formatted results on publish phase </li></ul>
  38. 39. Filter on format phase <ul><li>Only OK results are formatted as text format </li></ul>format: - module: Text filter: module: Status status: ok
  39. 40. Filter on notify phase <ul><li>Only NOK results are notified to IRC </li></ul>notify: - module: IRC filter: module: Status status: not ok
  40. 41. Filter on publish phase <ul><li>Text formatted results are displayed on a terminal,html formatted results are sent as a mail </li></ul>format: - module: Text - module: HTML pusblish: - module: Term filter: module: Type type: text/plain - module: Mail filter: module: Type type: text/html
  41. 42. Config validation
  42. 43. Config validation <ul><li>Kwalify – YAML schema validator http://www.kuwata-lab.com/kwalify / </li></ul><ul><li>Validate common config </li></ul><ul><li>Validate config of each plugin </li></ul><ul><li>Kwalify schemas for each plugin put under assets/kwalify/plugins </li></ul>
  43. 44. Mistake on common config test s : # test is right - module: Test::HTTP $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] - [/] Expected required key `test‘ - [/tests] Unexpected key `tests' at line 46
  44. 45. Mistake on plugin config test: - module: HTTP config: content s : XXX # content is right $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] Config error in Test::HTTP - [/contents] Unexpected key `contents' at line 66
  45. 46. Shell mode
  46. 47. Shell mode <ul><li>Execute shell command on hosts in config.yaml </li></ul>
  47. 48. Shell mode $ ./assurer.pl --shell assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [svn.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [ftp.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17
  48. 49. Exec on hosts belong to specified role $ ./assurer.pl --shell --role=web assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17
  49. 50. Special command !on assurer> !on app1.foo.com app2.foo.com do uptime # Exec on app1.foo.com and app2.foo.com only assurer> !on /.*.foo.com/ do uptime # Exec on hosts matched .*.foo.com only
  50. 51. Special command !with assurer> !with web db do uptime # exec on hosts belong to web role and db role assurer> !with /web|mail/ do uptime # exec on hosts belong to role match web or mail
  51. 52. Special command !test assurer> !test HTTP Assurer::Plugin::Test::HTTP [info] Testing HTTP on www0.mizzy.org ... ok 1 - HTTP status code of http://mizzy.org:80 is 200 not ok 2 – Content of http://mizzy.org:80 matches 'It works!‘ assurer>
  52. 53. Special command !test assurer> !test HTTP on app1.foo.com # test http on app1.foo.com assurer> !test HTTP on /.*.foo.com/ # can use regexp assurer> !test HTTP with web # test http on hosts belong to web role assurer> !test HTTP with /web|mail/ # can use regexp
  53. 54. Parallel testing
  54. 55. Pallarel testing <ul><li>Multiple test processes on test phase </li></ul><ul><li>Default is 5 </li></ul><ul><li>Can change number processes working at once </li></ul>$ assurer.pl --para=20
  55. 56. Distributed testing
  56. 57. Distributed testing localhost test0.mizzy.org www0.mizzy.org www1.mizzy.org www2.mizzy.org test1.mizzy.org test2.mizzy.org 1. ssh assurer_test.pl 3. Return result 2. Exec test
  57. 58. Config for distributed testing exec_on: - host: test0.mizzy.org priority: 3 - host: test1.mizzy.org priority: 2 - host: test2.mizzy.org priority: 1
  58. 59. Auto config generation
  59. 60. Auto config generation <ul><li>Scan hosts in template.yaml by Nmap::Scanner </li></ul><ul><li>Detect services working on the hosts and generete config.yaml automatically </li></ul><ul><li>config templates for each plugin put under assets/discover (http.yaml, mysql.yaml, etc) </li></ul># assurer.pl --discover -c template.yaml
  60. 61. Integragte Test::WWW::Mechanize
  61. 62. Integrate Test::WWW::Mechanize <ul><li>Can write tests using Test::WWW::Mechanize </li></ul><ul><li>Put .pl files under assets/plugins/Test-WWW-Mechanize </li></ul><ul><li>Pre-set $mech, $host on execution </li></ul>
  62. 63. asset file for Test::WWW::Mechanize # $host and $mech are set by Assure core $mech ->get_ok(&quot;http:// $host &quot;, &quot;got htttp:// $host &quot;); $mech ->content_contains('It works!', &quot;Content matches 'It works!'&quot;);
  63. 64. Inside Assurer
  64. 65. Assurer::Test module
  65. 66. Assurer::Test module <ul><li>Export Test::More like methods </li></ul><ul><li>is,() like(), ok() </li></ul><ul><li>Output test results formatted as TAP (Test Analyze Protocol) </li></ul>
  66. 67. Inside test plugins
  67. 68. Inside test plugins package Assurer::Plugin::Test::HTTP; use base qw( Assurer::Plugin::Test ); use Assurer::Test; sub register { my $self = shift; $self->register_tests( qw/ status content server / ); } sub status { my ( $self, $context, $args ) = @_; ... 中略 is( $res->code, '200', 'HTTP status code of $self->{url} is 200' ); }
  68. 69. Internal mechanism in test phase
  69. 70. Internal mechanism in test phase assurer.pl Assurer::bootstrap() Assurer::Dispatch::run() call exec directly or by ssh exec tests return test results call generate result objects next phase assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl Assurer::Result Assurer::Result Assurer::Result Assurer::Result Assurer::Result
  70. 71. Assurer::Dispatch::run() <ul><li>Exec assurer_test.pl for each test </li></ul><ul><li>Pallarelize by POE::Wheel::Run </li></ul><ul><li>Number of processed controlled by POE::Component::JobQueue </li></ul><ul><li>Pass config for plugin and context object serialized by YAML and encoded by Base64 </li></ul>
  71. 72. assurer_test.pl assurer_test.pl -– config = LS0tCmNvbmZpZzoKICBjb250ZW50OiBJ dCB3b3JrcyEKICBob3N0OiB3d3cubWl6enkub3JnCm1vZHVsZTogSFRUUApuYW1lOiBIVFRQIHRlc3QKcm9sZTogd2ViCg== - - context =LS0tICEhcGVybC9oYXNoOkFzc3VyZXIKYmFzZV9kaXI6IC9 ob21lL21peWEvc3ZrL0Fzc3VyZXIKY29uZmlnOgogIGZvcm1hdDoKICAgIC0gY29uZmlnOiAmMSB7fQogICAgICBtb2R1bGU6IFRleHQKICBnbG9iYWw6CiAgICBob3N0OiB+CiAgICBpbnRlcnZhbDogMwogICAgbG9nOgogICAgICBsZXZlbDogZGVidWcKICAgIG5vX2RpYWc6IDAKICAgIHJldHJ5OiAzCiAgaG9zdHM6CiAgICBmdHA6CiAgICAgIC0gZnRwLm1penp5Lm9yZwogICAgc3ZuOgogICAgICAtIHN2bi5taXp6eS5vcmcKICAgIHdlYjoKICAgICAgLSB3d3cubWl6enkub3JnCiAgICAgIC0gc3ZuLm1penp5Lm9yZwogICAgICAtIHRyYWMubWl6enkub3JnCiAgbm90aWZ5OgogICAgLSBjb25maWc6CiAgICAgICAgYW5ub3VuY2U6IG5vdGljZQogICAgICAgIGNoYXJzZXQ6IGlzby0yMDIyLWpwCiAgICAgICAgZGFlbW9uX3BvcnQ6IDk5OTEKICAgICAgICBuaWNrbmFtZTogYXNzdXJlcmJvdAog ... cont ...
  72. 73. Why POE::Wheel::Run ? <ul><li>Would like to exec tests as other processes </li></ul><ul><ul><li>Need non-blocking I/O with single process and POE </li></ul></ul><ul><ul><li>Can easily distribute to remote servers (exec directly on local, exec by ssh on remote ) </li></ul></ul><ul><li>Communication with test processes </li></ul><ul><ul><li>Can communicate easily with test processes by STDIN, STDOUT and STDERR </li></ul></ul>
  73. 74. Roadmap
  74. 75. Roadmap <ul><li>Integrate with Nagios/NRPE (Test::Nagios) </li></ul><ul><li>Enhance notify phase </li></ul><ul><li>Store Plugin (store test results in database) </li></ul><ul><li>Integrate with Archer, puppet (share host lists) </li></ul><ul><li>Test of Assurer itsself </li></ul>
  75. 76. Web site and IRC channels
  76. 77. Web sites and IRC channels <ul><li>http://assurer.jp/trac </li></ul><ul><li>http://mizzy.org/ </li></ul><ul><li>#assurer@chat.freenode.net </li></ul><ul><li>#assurer-ja@chat.freenode.net </li></ul>
  77. 78. Thanks
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×