Bring Your Smoke Tests to Life Michael Peters (mpeters@plusthree.com) http://sourceforge.net/projects/smolder

Problems Large and long running test suites? Code changes without test runs? Multiple languages How do you bring them all together? Project managers need to see pretty graphs?

Large and long running test suites?

Code changes without test runs?

Multiple languages

How do you bring them all together?

Project managers need to see pretty graphs?

Origins 5460 tests 28 minutes 5 different dev platforms PlusThree and PlunkIT

5460 tests

28 minutes

5 different dev platforms

Origins Automated build and test server emailed results every morning always huge (510k of HTML) developers had no control PlusThree and PlunkIT

Automated build and test server

emailed results every morning

always

huge (510k of HTML)

developers had no control

Origins PlusThree and Arcos Several variations on test runs 15,000+ tests 40+ minutes It was annoying and wasteful

Several variations on test runs

15,000+ tests

40+ minutes

It was annoying and wasteful

Origins Lot's of ideas from Pugs smoke server Developers customize Email frequency Email format Raw data Tests can be commented, categorized and invalidated Trends can be viewed Can host multiple projects public and private

Lot's of ideas from Pugs smoke server

Developers customize

Email frequency

Email format

Raw data

Tests can be commented, categorized and invalidated

Trends can be viewed

Can host multiple projects

public and private

Origins Built With Apache/mod_perl CGI::Application and family Template Toolkit SQLite (or MySQL if you prefer) Prototype.js (lots 'o AJAX) Full reports are lazy loaded

Apache/mod_perl

CGI::Application and family

Template Toolkit

SQLite (or MySQL if you prefer)

Prototype.js (lots 'o AJAX)

Full reports are lazy loaded

TAP Test Anything Protocol http://search.cpan.org/perldoc?TAP It's what test scripts output Test::Builder, Test::More, etc

Test Anything Protocol

http://search.cpan.org/perldoc?TAP

It's what test scripts output

Test::Builder, Test::More, etc

TAP 1..5 ok 1 – foo is fine ok 2 – testing bar value not ok 3 – The object isa Frobnicate # this is some useful # comment thing not ok 4 # SKIP Foo::Bar is not installed not ok 5 # TODO future feature

TAP use Test::More (tests => 5) ok($foo, 'foo is fine'); is($bar, 'right', 'testing bar value'); isa_ok($bar, 'Frobnicate'); diag(“this is some useful
comment thing”); SKIP: { skip('Foo::Bar is not installed' 1); ok($foo->bar, 'has a bar'); } TODO: { local $TODO = 'future feature'; ok($foo->feature, 'some feature'); }

TAP Started in Perl-land language-agnostic C PHP Python Java Javascript Parrot

Started in Perl-land

language-agnostic

C

PHP

Python

Java

Javascript

Parrot

TAP A single archive file (currently Tar) One file for each TAP stream A YAML file containing meta information about the test run Order Start and stop times Future expansion TAP Archives

A single archive file (currently Tar)

One file for each TAP stream

A YAML file containing meta information about the test run

Order

Start and stop times

Future expansion

Using How often? Every checkin? Periodically (every hour) ? Automated Tests

How often?

Every checkin?

Periodically (every hour) ?

Using Every Checkin SCM and build/test on same server checkin-hook script setup environment/services use TestRunner::Multi (module by Dave Rolsky) Automated Tests

Every Checkin

SCM and build/test on same server

checkin-hook script

setup environment/services

use TestRunner::Multi (module by Dave Rolsky)

Using Every Hour cron job setup environment/services Use TestRunner::Multi Automated Tests

Every Hour

cron job

setup environment/services

Use TestRunner::Multi

Future RSS/Atom feeds in addition to email Better integration with TestRunner::Multi Upcoming Features

RSS/Atom feeds in addition to email

Better integration with TestRunner::Multi

Future Code coverage reports and trends ???

Code coverage reports and trends

???