Assurer - a pluggable server testing/monitoring framework

  • 1,277 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,277
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Assurer – A server testing/monitoring framework Gosuke Miyashita http://mizzy.org/
  • 2. Myself
    • Known as Chocoboy Miyashita somewhere
    • Working in paperboy&co.
    • Mainly developed public/private web service APIs with perl and catalyt
    • ... but now also woking on servers and networks and so on
    • Today’s my son’s entrance ceremony of an elementary school
  • 3.  
  • 4. What’s Assurer ?
    • A server testing/monitoring framework like Plagger
      • Plugin mechanism
      • YAML configurable
      • assets
      • Test by Test::Base
    • Originally developed for one time test of servers that are newly built
    • Monitoring seems like testing, so decided to cover monitoring
  • 5. Where the name Assurer derived from ?
    • Pronounce “A-shu-ra”
    • Plagger like tools ended with “er”
      • Archer by tokuhirom
      • Observer by Hatena (not in public)
      • Dishuber by Yappo
      • Precure by Kan Fushihara
    • Testing is also called “Quality Assurance”
    MISS
  • 6. Image of exec Assurer #0
  • 7. Image of exec Assurer #1
  • 8. Image of Asura Baster
  • 9. Image of the real Asura
  • 10. Image of fake Asura
  • 11. Execution phases of Assurer
  • 12. Execution phases of Assurer
    • Test phase
      • Execute test
    • Nofity phase
      • Notify test results
    • Format phase
      • Format test results
    • Publish phase
      • Output formatted test results
  • 13. Relation of each phases Test Publish Format Nofity
  • 14. Details of each phases
  • 15. Test phase
  • 16. Test phase
    • Execute test
    • Plugins on this phase:
      • Test::HTTP
      • Test::SMTP
      • Test::DBI
      • Test::Ping
      • etc
  • 17. Config of test phase
    • test:
    • - module: HTTP
    • config:
    • host: www.mizzy.org
    • content: It works!
  • 18. Notify phase
  • 19. Notify フェーズ
    • Notify test results
    • Plugin on this phase:
      • Notify::IRC
  • 20. Config of notify phase
    • notify:
    • - module: IRC
    • config:
    • daemon_port: 9991
    • nickname: assurerbot
    • server_host: chat.freenode.net
    • server_port: 6667
    • server_channels:
    • - #assurer
    • - #assurer-jp
    • charset: iso-2022-jp
    • announce: notice
  • 21. Notify::IRC example
  • 22. Format phase
  • 23. Format phase
    • Format test results
    • Plugins on this phase:
      • Format::Text
      • Fomat::HTML
  • 24. Config of format phase
    • format:
    • - module: HTML
    • config:
    • css: /css/style.css
  • 25. Publish phase
  • 26. Publish phase
    • Output formatted results
    • Plugins on this phase:
      • Publish::Term
      • Publish::Mail
      • Publish::File
  • 27. Config of publish phase
    • publish:
    • - module: Mail
    • config:
    • subject: Test results from Assurer
    • to: someone@example.com
    • from: root@example.com
    publish: - module: Mail config: subject: Test results from Assurer to: someone@example.com from: root@example.com
  • 28. Format::Text + Publish::Term
  • 29. Format::HTML + Publish::File
  • 30. Format::Text + Publish::Mail
  • 31. Summary for now
    • Assurer is a serverPlagger
    • Test, Notify, Format and Publish phases
    • Easily change tests and outputs with plugins
  • 32. Extra functions
  • 33. Hosts and Roles
  • 34. Hosts and roles
    • Write target hosts and roles in one place on config.yaml
  • 35. Config without hosts and roles test: - module: HTTP config: host: www0.mizzy.org - module: HTTP config: host: www1.mizzy.org
  • 36. Config with hosts and roles test: - module: HTTP role: web hosts: web: - www0.mizzy.org - www1.mizzy.org
  • 37. Filter
  • 38. Filter
    • Filtering test results on format and notify phase
    • Filtering formatted results on publish phase
  • 39. Filter on format phase
    • Only OK results are formatted as text format
    format: - module: Text filter: module: Status status: ok
  • 40. Filter on notify phase
    • Only NOK results are notified to IRC
    notify: - module: IRC filter: module: Status status: not ok
  • 41. Filter on publish phase
    • Text formatted results are displayed on a terminal,html formatted results are sent as a mail
    format: - module: Text - module: HTML pusblish: - module: Term filter: module: Type type: text/plain - module: Mail filter: module: Type type: text/html
  • 42. Config validation
  • 43. Config validation
    • Kwalify – YAML schema validator http://www.kuwata-lab.com/kwalify /
    • Validate common config
    • Validate config of each plugin
    • Kwalify schemas for each plugin put under assets/kwalify/plugins
  • 44. Mistake on common config test s : # test is right - module: Test::HTTP $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] - [/] Expected required key `test‘ - [/tests] Unexpected key `tests' at line 46
  • 45. Mistake on plugin config test: - module: HTTP config: content s : XXX # content is right $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] Config error in Test::HTTP - [/contents] Unexpected key `contents' at line 66
  • 46. Shell mode
  • 47. Shell mode
    • Execute shell command on hosts in config.yaml
  • 48. Shell mode $ ./assurer.pl --shell assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [svn.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [ftp.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17
  • 49. Exec on hosts belong to specified role $ ./assurer.pl --shell --role=web assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17
  • 50. Special command !on assurer> !on app1.foo.com app2.foo.com do uptime # Exec on app1.foo.com and app2.foo.com only assurer> !on /.*.foo.com/ do uptime # Exec on hosts matched .*.foo.com only
  • 51. Special command !with assurer> !with web db do uptime # exec on hosts belong to web role and db role assurer> !with /web|mail/ do uptime # exec on hosts belong to role match web or mail
  • 52. Special command !test assurer> !test HTTP Assurer::Plugin::Test::HTTP [info] Testing HTTP on www0.mizzy.org ... ok 1 - HTTP status code of http://mizzy.org:80 is 200 not ok 2 – Content of http://mizzy.org:80 matches 'It works!‘ assurer>
  • 53. Special command !test assurer> !test HTTP on app1.foo.com # test http on app1.foo.com assurer> !test HTTP on /.*.foo.com/ # can use regexp assurer> !test HTTP with web # test http on hosts belong to web role assurer> !test HTTP with /web|mail/ # can use regexp
  • 54. Parallel testing
  • 55. Pallarel testing
    • Multiple test processes on test phase
    • Default is 5
    • Can change number processes working at once
    $ assurer.pl --para=20
  • 56. Distributed testing
  • 57. Distributed testing localhost test0.mizzy.org www0.mizzy.org www1.mizzy.org www2.mizzy.org test1.mizzy.org test2.mizzy.org 1. ssh assurer_test.pl 3. Return result 2. Exec test
  • 58. Config for distributed testing exec_on: - host: test0.mizzy.org priority: 3 - host: test1.mizzy.org priority: 2 - host: test2.mizzy.org priority: 1
  • 59. Auto config generation
  • 60. Auto config generation
    • Scan hosts in template.yaml by Nmap::Scanner
    • Detect services working on the hosts and generete config.yaml automatically
    • config templates for each plugin put under assets/discover (http.yaml, mysql.yaml, etc)
    # assurer.pl --discover -c template.yaml
  • 61. Integragte Test::WWW::Mechanize
  • 62. Integrate Test::WWW::Mechanize
    • Can write tests using Test::WWW::Mechanize
    • Put .pl files under assets/plugins/Test-WWW-Mechanize
    • Pre-set $mech, $host on execution
  • 63. asset file for Test::WWW::Mechanize # $host and $mech are set by Assure core $mech ->get_ok("http:// $host ", "got htttp:// $host "); $mech ->content_contains('It works!', "Content matches 'It works!'");
  • 64. Inside Assurer
  • 65. Assurer::Test module
  • 66. Assurer::Test module
    • Export Test::More like methods
    • is,() like(), ok()
    • Output test results formatted as TAP (Test Analyze Protocol)
  • 67. Inside test plugins
  • 68. Inside test plugins package Assurer::Plugin::Test::HTTP; use base qw( Assurer::Plugin::Test ); use Assurer::Test; sub register { my $self = shift; $self->register_tests( qw/ status content server / ); } sub status { my ( $self, $context, $args ) = @_; ... 中略 is( $res->code, '200', 'HTTP status code of $self->{url} is 200' ); }
  • 69. Internal mechanism in test phase
  • 70. Internal mechanism in test phase assurer.pl Assurer::bootstrap() Assurer::Dispatch::run() call exec directly or by ssh exec tests return test results call generate result objects next phase assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl Assurer::Result Assurer::Result Assurer::Result Assurer::Result Assurer::Result
  • 71. Assurer::Dispatch::run()
    • Exec assurer_test.pl for each test
    • Pallarelize by POE::Wheel::Run
    • Number of processed controlled by POE::Component::JobQueue
    • Pass config for plugin and context object serialized by YAML and encoded by Base64
  • 72. assurer_test.pl assurer_test.pl -– config = LS0tCmNvbmZpZzoKICBjb250ZW50OiBJ dCB3b3JrcyEKICBob3N0OiB3d3cubWl6enkub3JnCm1vZHVsZTogSFRUUApuYW1lOiBIVFRQIHRlc3QKcm9sZTogd2ViCg== - - context =LS0tICEhcGVybC9oYXNoOkFzc3VyZXIKYmFzZV9kaXI6IC9 ob21lL21peWEvc3ZrL0Fzc3VyZXIKY29uZmlnOgogIGZvcm1hdDoKICAgIC0gY29uZmlnOiAmMSB7fQogICAgICBtb2R1bGU6IFRleHQKICBnbG9iYWw6CiAgICBob3N0OiB+CiAgICBpbnRlcnZhbDogMwogICAgbG9nOgogICAgICBsZXZlbDogZGVidWcKICAgIG5vX2RpYWc6IDAKICAgIHJldHJ5OiAzCiAgaG9zdHM6CiAgICBmdHA6CiAgICAgIC0gZnRwLm1penp5Lm9yZwogICAgc3ZuOgogICAgICAtIHN2bi5taXp6eS5vcmcKICAgIHdlYjoKICAgICAgLSB3d3cubWl6enkub3JnCiAgICAgIC0gc3ZuLm1penp5Lm9yZwogICAgICAtIHRyYWMubWl6enkub3JnCiAgbm90aWZ5OgogICAgLSBjb25maWc6CiAgICAgICAgYW5ub3VuY2U6IG5vdGljZQogICAgICAgIGNoYXJzZXQ6IGlzby0yMDIyLWpwCiAgICAgICAgZGFlbW9uX3BvcnQ6IDk5OTEKICAgICAgICBuaWNrbmFtZTogYXNzdXJlcmJvdAog ... cont ...
  • 73. Why POE::Wheel::Run ?
    • Would like to exec tests as other processes
      • Need non-blocking I/O with single process and POE
      • Can easily distribute to remote servers (exec directly on local, exec by ssh on remote )
    • Communication with test processes
      • Can communicate easily with test processes by STDIN, STDOUT and STDERR
  • 74. Roadmap
  • 75. Roadmap
    • Integrate with Nagios/NRPE (Test::Nagios)
    • Enhance notify phase
    • Store Plugin (store test results in database)
    • Integrate with Archer, puppet (share host lists)
    • Test of Assurer itsself
  • 76. Web site and IRC channels
  • 77. Web sites and IRC channels
    • http://assurer.jp/trac
    • http://mizzy.org/
    • #assurer@chat.freenode.net
    • #assurer-ja@chat.freenode.net
  • 78. Thanks