Assurer - a pluggable server testing/monitoring framework
Upcoming SlideShare
Loading in...5
×
 

Assurer - a pluggable server testing/monitoring framework

on

  • 1,701 views

 

Statistics

Views

Total Views
1,701
Views on SlideShare
1,584
Embed Views
117

Actions

Likes
0
Downloads
2
Comments
0

1 Embed 117

http://coderwall.com 117

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Assurer - a pluggable server testing/monitoring framework Assurer - a pluggable server testing/monitoring framework Presentation Transcript

  • Assurer – A server testing/monitoring framework Gosuke Miyashita http://mizzy.org/
  • Myself
    • Known as Chocoboy Miyashita somewhere
    • Working in paperboy&co.
    • Mainly developed public/private web service APIs with perl and catalyt
    • ... but now also woking on servers and networks and so on
    • Today’s my son’s entrance ceremony of an elementary school
  •  
  • What’s Assurer ?
    • A server testing/monitoring framework like Plagger
      • Plugin mechanism
      • YAML configurable
      • assets
      • Test by Test::Base
    • Originally developed for one time test of servers that are newly built
    • Monitoring seems like testing, so decided to cover monitoring
  • Where the name Assurer derived from ?
    • Pronounce “A-shu-ra”
    • Plagger like tools ended with “er”
      • Archer by tokuhirom
      • Observer by Hatena (not in public)
      • Dishuber by Yappo
      • Precure by Kan Fushihara
    • Testing is also called “Quality Assurance”
    MISS
  • Image of exec Assurer #0
  • Image of exec Assurer #1
  • Image of Asura Baster
  • Image of the real Asura
  • Image of fake Asura
  • Execution phases of Assurer
  • Execution phases of Assurer
    • Test phase
      • Execute test
    • Nofity phase
      • Notify test results
    • Format phase
      • Format test results
    • Publish phase
      • Output formatted test results
  • Relation of each phases Test Publish Format Nofity
  • Details of each phases
  • Test phase
  • Test phase
    • Execute test
    • Plugins on this phase:
      • Test::HTTP
      • Test::SMTP
      • Test::DBI
      • Test::Ping
      • etc
  • Config of test phase
    • test:
    • - module: HTTP
    • config:
    • host: www.mizzy.org
    • content: It works!
  • Notify phase
  • Notify フェーズ
    • Notify test results
    • Plugin on this phase:
      • Notify::IRC
  • Config of notify phase
    • notify:
    • - module: IRC
    • config:
    • daemon_port: 9991
    • nickname: assurerbot
    • server_host: chat.freenode.net
    • server_port: 6667
    • server_channels:
    • - #assurer
    • - #assurer-jp
    • charset: iso-2022-jp
    • announce: notice
  • Notify::IRC example
  • Format phase
  • Format phase
    • Format test results
    • Plugins on this phase:
      • Format::Text
      • Fomat::HTML
  • Config of format phase
    • format:
    • - module: HTML
    • config:
    • css: /css/style.css
  • Publish phase
  • Publish phase
    • Output formatted results
    • Plugins on this phase:
      • Publish::Term
      • Publish::Mail
      • Publish::File
  • Config of publish phase
    • publish:
    • - module: Mail
    • config:
    • subject: Test results from Assurer
    • to: someone@example.com
    • from: root@example.com
    publish: - module: Mail config: subject: Test results from Assurer to: someone@example.com from: root@example.com
  • Format::Text + Publish::Term
  • Format::HTML + Publish::File
  • Format::Text + Publish::Mail
  • Summary for now
    • Assurer is a serverPlagger
    • Test, Notify, Format and Publish phases
    • Easily change tests and outputs with plugins
  • Extra functions
  • Hosts and Roles
  • Hosts and roles
    • Write target hosts and roles in one place on config.yaml
  • Config without hosts and roles test: - module: HTTP config: host: www0.mizzy.org - module: HTTP config: host: www1.mizzy.org
  • Config with hosts and roles test: - module: HTTP role: web hosts: web: - www0.mizzy.org - www1.mizzy.org
  • Filter
  • Filter
    • Filtering test results on format and notify phase
    • Filtering formatted results on publish phase
  • Filter on format phase
    • Only OK results are formatted as text format
    format: - module: Text filter: module: Status status: ok
  • Filter on notify phase
    • Only NOK results are notified to IRC
    notify: - module: IRC filter: module: Status status: not ok
  • Filter on publish phase
    • Text formatted results are displayed on a terminal,html formatted results are sent as a mail
    format: - module: Text - module: HTML pusblish: - module: Term filter: module: Type type: text/plain - module: Mail filter: module: Type type: text/html
  • Config validation
  • Config validation
    • Kwalify – YAML schema validator http://www.kuwata-lab.com/kwalify /
    • Validate common config
    • Validate config of each plugin
    • Kwalify schemas for each plugin put under assets/kwalify/plugins
  • Mistake on common config test s : # test is right - module: Test::HTTP $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] - [/] Expected required key `test‘ - [/tests] Unexpected key `tests' at line 46
  • Mistake on plugin config test: - module: HTTP config: content s : XXX # content is right $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] Config error in Test::HTTP - [/contents] Unexpected key `contents' at line 66
  • Shell mode
  • Shell mode
    • Execute shell command on hosts in config.yaml
  • Shell mode $ ./assurer.pl --shell assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [svn.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [ftp.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17
  • Exec on hosts belong to specified role $ ./assurer.pl --shell --role=web assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17
  • Special command !on assurer> !on app1.foo.com app2.foo.com do uptime # Exec on app1.foo.com and app2.foo.com only assurer> !on /.*.foo.com/ do uptime # Exec on hosts matched .*.foo.com only
  • Special command !with assurer> !with web db do uptime # exec on hosts belong to web role and db role assurer> !with /web|mail/ do uptime # exec on hosts belong to role match web or mail
  • Special command !test assurer> !test HTTP Assurer::Plugin::Test::HTTP [info] Testing HTTP on www0.mizzy.org ... ok 1 - HTTP status code of http://mizzy.org:80 is 200 not ok 2 – Content of http://mizzy.org:80 matches 'It works!‘ assurer>
  • Special command !test assurer> !test HTTP on app1.foo.com # test http on app1.foo.com assurer> !test HTTP on /.*.foo.com/ # can use regexp assurer> !test HTTP with web # test http on hosts belong to web role assurer> !test HTTP with /web|mail/ # can use regexp
  • Parallel testing
  • Pallarel testing
    • Multiple test processes on test phase
    • Default is 5
    • Can change number processes working at once
    $ assurer.pl --para=20
  • Distributed testing
  • Distributed testing localhost test0.mizzy.org www0.mizzy.org www1.mizzy.org www2.mizzy.org test1.mizzy.org test2.mizzy.org 1. ssh assurer_test.pl 3. Return result 2. Exec test
  • Config for distributed testing exec_on: - host: test0.mizzy.org priority: 3 - host: test1.mizzy.org priority: 2 - host: test2.mizzy.org priority: 1
  • Auto config generation
  • Auto config generation
    • Scan hosts in template.yaml by Nmap::Scanner
    • Detect services working on the hosts and generete config.yaml automatically
    • config templates for each plugin put under assets/discover (http.yaml, mysql.yaml, etc)
    # assurer.pl --discover -c template.yaml
  • Integragte Test::WWW::Mechanize
  • Integrate Test::WWW::Mechanize
    • Can write tests using Test::WWW::Mechanize
    • Put .pl files under assets/plugins/Test-WWW-Mechanize
    • Pre-set $mech, $host on execution
  • asset file for Test::WWW::Mechanize # $host and $mech are set by Assure core $mech ->get_ok("http:// $host ", "got htttp:// $host "); $mech ->content_contains('It works!', "Content matches 'It works!'");
  • Inside Assurer
  • Assurer::Test module
  • Assurer::Test module
    • Export Test::More like methods
    • is,() like(), ok()
    • Output test results formatted as TAP (Test Analyze Protocol)
  • Inside test plugins
  • Inside test plugins package Assurer::Plugin::Test::HTTP; use base qw( Assurer::Plugin::Test ); use Assurer::Test; sub register { my $self = shift; $self->register_tests( qw/ status content server / ); } sub status { my ( $self, $context, $args ) = @_; ... 中略 is( $res->code, '200', 'HTTP status code of $self->{url} is 200' ); }
  • Internal mechanism in test phase
  • Internal mechanism in test phase assurer.pl Assurer::bootstrap() Assurer::Dispatch::run() call exec directly or by ssh exec tests return test results call generate result objects next phase assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl Assurer::Result Assurer::Result Assurer::Result Assurer::Result Assurer::Result
  • Assurer::Dispatch::run()
    • Exec assurer_test.pl for each test
    • Pallarelize by POE::Wheel::Run
    • Number of processed controlled by POE::Component::JobQueue
    • Pass config for plugin and context object serialized by YAML and encoded by Base64
  • assurer_test.pl assurer_test.pl -– config = LS0tCmNvbmZpZzoKICBjb250ZW50OiBJ dCB3b3JrcyEKICBob3N0OiB3d3cubWl6enkub3JnCm1vZHVsZTogSFRUUApuYW1lOiBIVFRQIHRlc3QKcm9sZTogd2ViCg== - - context =LS0tICEhcGVybC9oYXNoOkFzc3VyZXIKYmFzZV9kaXI6IC9 ob21lL21peWEvc3ZrL0Fzc3VyZXIKY29uZmlnOgogIGZvcm1hdDoKICAgIC0gY29uZmlnOiAmMSB7fQogICAgICBtb2R1bGU6IFRleHQKICBnbG9iYWw6CiAgICBob3N0OiB+CiAgICBpbnRlcnZhbDogMwogICAgbG9nOgogICAgICBsZXZlbDogZGVidWcKICAgIG5vX2RpYWc6IDAKICAgIHJldHJ5OiAzCiAgaG9zdHM6CiAgICBmdHA6CiAgICAgIC0gZnRwLm1penp5Lm9yZwogICAgc3ZuOgogICAgICAtIHN2bi5taXp6eS5vcmcKICAgIHdlYjoKICAgICAgLSB3d3cubWl6enkub3JnCiAgICAgIC0gc3ZuLm1penp5Lm9yZwogICAgICAtIHRyYWMubWl6enkub3JnCiAgbm90aWZ5OgogICAgLSBjb25maWc6CiAgICAgICAgYW5ub3VuY2U6IG5vdGljZQogICAgICAgIGNoYXJzZXQ6IGlzby0yMDIyLWpwCiAgICAgICAgZGFlbW9uX3BvcnQ6IDk5OTEKICAgICAgICBuaWNrbmFtZTogYXNzdXJlcmJvdAog ... cont ...
  • Why POE::Wheel::Run ?
    • Would like to exec tests as other processes
      • Need non-blocking I/O with single process and POE
      • Can easily distribute to remote servers (exec directly on local, exec by ssh on remote )
    • Communication with test processes
      • Can communicate easily with test processes by STDIN, STDOUT and STDERR
  • Roadmap
  • Roadmap
    • Integrate with Nagios/NRPE (Test::Nagios)
    • Enhance notify phase
    • Store Plugin (store test results in database)
    • Integrate with Archer, puppet (share host lists)
    • Test of Assurer itsself
  • Web site and IRC channels
  • Web sites and IRC channels
    • http://assurer.jp/trac
    • http://mizzy.org/
    • #assurer@chat.freenode.net
    • #assurer-ja@chat.freenode.net
  • Thanks