Reliable acceptance testing
Upcoming SlideShare
Loading in...5
×
 

Reliable acceptance testing

on

  • 1,132 views

 

Statistics

Views

Total Views
1,132
Views on SlideShare
1,132
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Reliable acceptance testing Reliable acceptance testing Presentation Transcript

  • Dagfinn Reiersøl, ABC Startsiden 1 Reliable acceptance testing
  • Dagfinn Reiersøl, ABC Startsiden 2 Manual testing is immoral • Not what humans are good at • Time consuming = expensive • High stress • Error-prone • Exception: exploratory testing
  • Dagfinn Reiersøl, ABC Startsiden 3 Types of automated testing • Unit tests: clean code • Acceptance tests: communicate with user / customer • Other functional tests: ensure that everything works
  • Dagfinn Reiersøl, ABC Startsiden 4 High-level challenges of acceptance testing • Getting the Product Owner involved • Test Doubles • Not over-testing the user interface
  • Dagfinn Reiersøl, ABC Startsiden 5 Unit tests • Keep code clean • Avoid “hidden” bugs • Aid design • Test-driven development
  • Dagfinn Reiersøl, ABC Startsiden 6 Scrum Product Owner • Represents customer(s) and other stakeholders • Owns the product backlog • Is not a developer
  • Dagfinn Reiersøl, ABC Startsiden 7 Acceptance tests • For the Product Owner or possibly testers • Goal: Make sure we implement the right features • Ideally editable by a non-programmer • Tests features, not modules • Make sure you're done
  • Dagfinn Reiersøl, ABC Startsiden 8 Other functional tests • Goal: Make sure the implementation works • Could be testing the same thing as acceptance tests, but • need not be business-readable
  • Dagfinn Reiersøl, ABC Startsiden 9 The Test Double pattern
  • Dagfinn Reiersøl, ABC Startsiden 10 Surrounded by “Test Doubles”
  • Dagfinn Reiersøl, ABC Startsiden 11 Web testing: advantages • You can test exactly what the user sees • There are tools to test JavaScript-heavy / Ajax apps • There are tools to record and play back tests • “It seemed like an good idea at the time”
  • Dagfinn Reiersøl, ABC Startsiden 12 Web testing: disadvantages • Fragile: Small UI changes break tests • Fragile: External services break tests • Slow • Harder to replace external services with fakes
  • Dagfinn Reiersøl, ABC Startsiden 13 Strategies for robustness • Test Bus • Test patterns • Test refactoring • Making tests unnecessary
  • Dagfinn Reiersøl, ABC Startsiden 14 Web testing: essentials • Must be in a programming language (conditionals, variables) • Language not necessarily PHP, since HTTP allows communication between languages • “Business-readable” programming language (FitNesse wiki, Gherkin...)
  • Dagfinn Reiersøl, ABC Startsiden 15 Web testing tips • Test the HTML output using regex, XPath, CSS selector • Use element IDs or names to test links, forms and fields • Log HTTP requests in the application
  • Dagfinn Reiersøl, ABC Startsiden 16 How to use the request log Automated test fails but not manual run from browser • Run the automated test, find the failing request in the log. • Run the app manually, find the successful request in the log. • Compare.
  • Dagfinn Reiersøl, ABC Startsiden 17 Using SimpleTest <?php require_once 'simpletest/autorun.php'; require_once 'simpletest/web_tester.php'; error_reporting (E_ALL); class JoomlaSimpleTestTest extends WebTestCase { function testJoomla() { $this->get('http://localhost/joomla'); $this->assertResponse(200); $this->assertTitle('Welcome to the Frontpage'); $this->assertText('Welcome to the Frontpage'); $this->click('Welcome to Joomla!'); $this->assertPattern('/Joomla! is a free.*content publishing system/is'); } }
  • Dagfinn Reiersøl, ABC Startsiden 18 Using PHPUnit with Selenium <?php require_once 'PHPUnit/Extensions/SeleniumTestCase.php'; class JoomlaPhpUnitTest extends PHPUnit_Extensions_SeleniumTestCase { protected function setUp(){ $this->setBrowser('*firefox'); $this->setBrowserUrl('http://localhost/'); } public function testJoomla() { $this->open('http://localhost/joomla/'); $this->assertTitle('Welcome to the Frontpage'); $this->assertTextPresent('Welcome to the Frontpage'); $this ->clickAndWait( '//ul[@class='latestnews']/li[4]/a'); $this ->assertTextPresent('Joomla! is a free open'); } }
  • Dagfinn Reiersøl, ABC Startsiden 19 SimpleTest vs. PHPUnit / Selenium Simpletest: – Faster – Easier to use PHPUnit with Selenium – Heavier but more advanced – Can test JavaScript-heavy / Ajax apps – Can test cross-browser compatibility
  • Dagfinn Reiersøl, ABC Startsiden 20 Don't overdo Selenium “He ate a lot of brazil nuts which is a big deal because they contain selenium, which in high doses causes fatigue, vomiting, skin irritation, discharge from the fingernail beds and hair loss.” - House, M.D.
  • Dagfinn Reiersøl, ABC Startsiden 21 Acceptance test frameworks • FitNesse: has specific support for PHP • Cucumber: has PHP-related documentation • Robot Framework • Concordion
  • Dagfinn Reiersøl, ABC Startsiden 22 Using FitNesse with SimpleTest
  • Dagfinn Reiersøl, ABC Startsiden 23 PhpSlim Fixture for FitNesse test (simplistic!) <?php require_once 'simpletest/browser.php'; class SimpleTestFixture { function __construct() { $this->browser = new SimpleBrowser; } function goToPage($url) { $this->content = $this->browser->get($url); return !empty($this->content); } function pattern($pattern) { return preg_match("/$pattern/is",$this->content) > 0; } }
  • Dagfinn Reiersøl, ABC Startsiden 24 Using Cucumber with Webrat • joomla.feature (Cucumber Gherkin business-readable) • webrat_steps (Ruby code testing via HTTP) Feature: Joomla welcome In order to understand what Joomla is As a potential user I want to be able to read an introduction to Joomla Scenario: Welcome page Given I am on the home page When I follow "Welcome to Joomla!" Then I should see "Joomla! is a"
  • Dagfinn Reiersøl, ABC Startsiden 25 Webrat steps for Cucumber • Ruby code runs Webrat which runs the web app using HTTP Given /^I am on (.+)$/ do |page_name|   visit path_to(page_name) end When /^I follow "([^"]*)"$/ do |link|   click_link (link) end  Then /^I should see "([^"]*)"$/ do |text|   response_body.should contain(text) end
  • Dagfinn Reiersøl, ABC Startsiden 26 Avoiding difficult tests • It's hard to test behaviors that cut across the HTTP request • Improve architecture so these behaviors can't fail • Example: duplication between input control name and HTTP variable name
  • Dagfinn Reiersøl, ABC Startsiden 27 Hard to test: name duplication <input type="text" name="address1" value="<?= $address1 ?>"> // Oops... echo $_POST['address_1'];
  • Dagfinn Reiersøl, ABC Startsiden 28 Getting rid of name duplication • Use a form handling package • Or do it yourself. Simplistic version here: <?php define('ADDRESS1_NAME', 'address1'); $address1_value = $_POST[ADDRESS1_NAME]; <html> ... <input name="<?= ADDRESS1_NAME ?>" value="<?= $address1_value ?>">
  • Dagfinn Reiersøl, ABC Startsiden 29 Refactoring tests • Refactor or be doomed to drown in duplicate code • Especially true of web tests • Most important: eliminate duplication
  • Dagfinn Reiersøl, ABC Startsiden 30 BDD-style test names class JoomlaSimpleTestTest extends WebTestCase {     function setUp() {         $this­>get('http://localhost/joomla');     }     function testFrontPageIsValid() {         $this­>assertResponse(200);         $this­>assertTitle('Welcome to the Frontpage');         $this­>assertText('Welcome to the Frontpage');     }     function testWelcomePageIsValid() {         $this­>click('Welcome to Joomla!');         $this­>assertPattern('/Joomla! is a free.*content  publishing system/is');     } }
  • Dagfinn Reiersøl, ABC Startsiden 31 Using custom assertions instead class JoomlaSimpleTestTest extends WebTestCase {     function testJoomla() {         $this­>get('http://localhost/joomla');         $this­>assertValidHomePage();         $this­>click('Welcome to Joomla!');         $this­>assertValidWelcomePage();     }     function assertValidHomePage() {         $msg = 'Error on home page';         $this­>assertResponse(200,$msg);         $this­>assertTitle('Welcome to the Frontpage',$msg);         $this­>assertText('Welcome to the Fontpage',$msg);     }     function assertValidWelcomePage() {         $this­>assertPattern(            '/Joomla! is a free.*content publishing system/is',            'Error on welcome page');     }
  • Dagfinn Reiersøl, ABC Startsiden 32 ComposedRegex http://martinfowler.com/bliki/ComposedRegex.html • This regex tolerates markup (or anything else) between the content items • Yes, I know the email regex is simplistic $email = '.*w+@w+.w+.*';         $firstname = '.*w.*';         $lastname = '.*w.*';         $date = '.*20dd­dd­dd( |&nbsp;|T)dd:dd.*';         $regex = 'Email address:'. $email .                   'Name: ' . $firstname . $lastname .                   'Date:' . $date;
  • Dagfinn Reiersøl, ABC Startsiden 33 Re-using setup • Refactor gradually as needed: – When test class gets big, split it and... – ...extract setup method into a separate class – When setup is common to many tests... – ...extract setup class into a separate file class JoomlaTestSetup {     function setUp() {         $this­>get('http://localhost/joomla');     } } class JoomlaSimpleTestTest extends JoomlaTestSetup..
  • Dagfinn Reiersøl, ABC Startsiden 34 Useful types of classes in web testing • Scraper • Domain Object • Specification
  • Dagfinn Reiersøl, ABC Startsiden 35 Domain object test version class UserTest extends WebTestCase {     function setUp() {         $this­>user = new TestUser(             'Jane','Doe','jane@example.com');     }     function testCanRegisterUser...         $this­>setFieldByName(             'firstname',            $this­>user­>firstname         );
  • Dagfinn Reiersøl, ABC Startsiden 36 Given-when-then in PHP class JoomlaSimpleTestTest extends WebTestCase {     function setUp() {         $this­>pages = array(             'JoomlaHomePage' => 'http://localhost/joomla');     }     function testWelcomPageIsValid() {         $this­>givenIAmOn('JoomlaHomePage');         $this­>whenIFollow('Welcome to Joomla!');         $this­>thenIShouldSee('Joomla! is a free');     }     function givenIAmOn($name) {         $this­>get($this­>pages[$name]);     }     function whenIFollow($text) {        $this­>clickLink($text);      }     function thenIShouldSee($pattern) {         $this­>assertPattern("/$pattern/is");     }
  • Dagfinn Reiersøl, ABC Startsiden 37 Test bus http://www.martinfowler.com/ieeeSoftware/testBus.pdf
  • Dagfinn Reiersøl, ABC Startsiden 38 Recommendations from Uncle Bob • Test the features through the test bus • Test the user interface separately in isolation • Run a few end-to-end tests to test the “plumbing”
  • Dagfinn Reiersøl, ABC Startsiden 39 Test bus Where is the test bus? Some possbilities: • Test the Model • Test the Controller • Test the server side of an Ajax or “thin server” application • Test a public API
  • Dagfinn Reiersøl, ABC Startsiden 40 Test bus with presentation tests
  • Dagfinn Reiersøl, ABC Startsiden 41 Fake web service • Easier to do when not testing via HTTP • Sensing/verification is very hard to do
  • Dagfinn Reiersøl, ABC Startsiden 42 Self-initializing fake http://martinfowler.com/bliki/SelfInitializingFake.html
  • Dagfinn Reiersøl, ABC Startsiden 43 Self-initializing fake http://martinfowler.com/bliki/SelfInitializingFake.html