Dagfinn Reiersøl, ABC Startsiden 1
Reliable acceptance testing
Dagfinn Reiersøl, ABC Startsiden 2
Manual testing is immoral
• Not what humans are good at
• Time consuming = expensive
• ...
Dagfinn Reiersøl, ABC Startsiden 3
Types of automated testing
• Unit tests: clean code
• Acceptance tests: communicate wit...
Dagfinn Reiersøl, ABC Startsiden 4
High-level challenges of
acceptance testing
• Getting the Product Owner involved
• Test...
Dagfinn Reiersøl, ABC Startsiden 5
Unit tests
• Keep code clean
• Avoid “hidden” bugs
• Aid design
• Test-driven developme...
Dagfinn Reiersøl, ABC Startsiden 6
Scrum Product Owner
• Represents customer(s) and other stakeholders
• Owns the product ...
Dagfinn Reiersøl, ABC Startsiden 7
Acceptance tests
• For the Product Owner or possibly testers
• Goal: Make sure we imple...
Dagfinn Reiersøl, ABC Startsiden 8
Other functional tests
• Goal: Make sure the implementation works
• Could be testing th...
Dagfinn Reiersøl, ABC Startsiden 9
The Test Double pattern
Dagfinn Reiersøl, ABC Startsiden 10
Surrounded by “Test Doubles”
Dagfinn Reiersøl, ABC Startsiden 11
Web testing: advantages
• You can test exactly what the user sees
• There are tools to...
Dagfinn Reiersøl, ABC Startsiden 12
Web testing: disadvantages
• Fragile: Small UI changes break tests
• Fragile: External...
Dagfinn Reiersøl, ABC Startsiden 13
Strategies for robustness
• Test Bus
• Test patterns
• Test refactoring
• Making tests...
Dagfinn Reiersøl, ABC Startsiden 14
Web testing: essentials
• Must be in a programming language (conditionals, variables)
...
Dagfinn Reiersøl, ABC Startsiden 15
Web testing tips
• Test the HTML output using regex, XPath, CSS selector
• Use element...
Dagfinn Reiersøl, ABC Startsiden 16
How to use the request log
Automated test fails but not manual run from browser
• Run ...
Dagfinn Reiersøl, ABC Startsiden 17
Using SimpleTest
<?php
require_once 'simpletest/autorun.php';
require_once 'simpletest...
Dagfinn Reiersøl, ABC Startsiden 18
Using PHPUnit with Selenium
<?php
require_once 'PHPUnit/Extensions/SeleniumTestCase.ph...
Dagfinn Reiersøl, ABC Startsiden 19
SimpleTest vs. PHPUnit /
Selenium
Simpletest:
– Faster
– Easier to use
PHPUnit with Se...
Dagfinn Reiersøl, ABC Startsiden 20
Don't overdo Selenium
“He ate a lot of brazil nuts which is a big deal because they
co...
Dagfinn Reiersøl, ABC Startsiden 21
Acceptance test frameworks
• FitNesse: has specific support for PHP
• Cucumber: has PH...
Dagfinn Reiersøl, ABC Startsiden 22
Using FitNesse with SimpleTest
Dagfinn Reiersøl, ABC Startsiden 23
PhpSlim Fixture for FitNesse test
(simplistic!)
<?php
require_once 'simpletest/browser...
Dagfinn Reiersøl, ABC Startsiden 24
Using Cucumber with Webrat
• joomla.feature (Cucumber Gherkin business-readable)
• web...
Dagfinn Reiersøl, ABC Startsiden 25
Webrat steps for Cucumber
• Ruby code runs Webrat which runs the web app using HTTP
Gi...
Dagfinn Reiersøl, ABC Startsiden 26
Avoiding difficult tests
• It's hard to test behaviors that cut across the HTTP reques...
Dagfinn Reiersøl, ABC Startsiden 27
Hard to test: name duplication
<input type="text" name="address1"
value="<?= $address1...
Dagfinn Reiersøl, ABC Startsiden 28
Getting rid of name duplication
• Use a form handling package
• Or do it yourself. Sim...
Dagfinn Reiersøl, ABC Startsiden 29
Refactoring tests
• Refactor or be doomed to drown in duplicate code
• Especially true...
Dagfinn Reiersøl, ABC Startsiden 30
BDD-style test names
class JoomlaSimpleTestTest extends WebTestCase {
    function set...
Dagfinn Reiersøl, ABC Startsiden 31
Using custom assertions instead
class JoomlaSimpleTestTest extends WebTestCase {
    f...
Dagfinn Reiersøl, ABC Startsiden 32
ComposedRegex
http://martinfowler.com/bliki/ComposedRegex.html
• This regex tolerates ...
Dagfinn Reiersøl, ABC Startsiden 33
Re-using setup
• Refactor gradually as needed:
– When test class gets big, split it an...
Dagfinn Reiersøl, ABC Startsiden 34
Useful types of classes in web
testing
• Scraper
• Domain Object
• Specification
Dagfinn Reiersøl, ABC Startsiden 35
Domain object test version
class UserTest extends WebTestCase {
    function setUp() {...
Dagfinn Reiersøl, ABC Startsiden 36
Given-when-then in PHP
class JoomlaSimpleTestTest extends WebTestCase {
    function s...
Dagfinn Reiersøl, ABC Startsiden 37
Test bus
http://www.martinfowler.com/ieeeSoftware/testBus.pdf
Dagfinn Reiersøl, ABC Startsiden 38
Recommendations from Uncle
Bob
• Test the features through the test bus
• Test the use...
Dagfinn Reiersøl, ABC Startsiden 39
Test bus
Where is the test bus? Some possbilities:
• Test the Model
• Test the Control...
Dagfinn Reiersøl, ABC Startsiden 40
Test bus with presentation tests
Dagfinn Reiersøl, ABC Startsiden 41
Fake web service
• Easier to do when not
testing via HTTP
• Sensing/verification is
ve...
Dagfinn Reiersøl, ABC Startsiden 42
Self-initializing fake
http://martinfowler.com/bliki/SelfInitializingFake.html
Dagfinn Reiersøl, ABC Startsiden 43
Self-initializing fake
http://martinfowler.com/bliki/SelfInitializingFake.html
Upcoming SlideShare
Loading in...5
×

Reliable acceptance testing

1,418

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,418
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Reliable acceptance testing

  1. 1. Dagfinn Reiersøl, ABC Startsiden 1 Reliable acceptance testing
  2. 2. Dagfinn Reiersøl, ABC Startsiden 2 Manual testing is immoral • Not what humans are good at • Time consuming = expensive • High stress • Error-prone • Exception: exploratory testing
  3. 3. Dagfinn Reiersøl, ABC Startsiden 3 Types of automated testing • Unit tests: clean code • Acceptance tests: communicate with user / customer • Other functional tests: ensure that everything works
  4. 4. Dagfinn Reiersøl, ABC Startsiden 4 High-level challenges of acceptance testing • Getting the Product Owner involved • Test Doubles • Not over-testing the user interface
  5. 5. Dagfinn Reiersøl, ABC Startsiden 5 Unit tests • Keep code clean • Avoid “hidden” bugs • Aid design • Test-driven development
  6. 6. Dagfinn Reiersøl, ABC Startsiden 6 Scrum Product Owner • Represents customer(s) and other stakeholders • Owns the product backlog • Is not a developer
  7. 7. Dagfinn Reiersøl, ABC Startsiden 7 Acceptance tests • For the Product Owner or possibly testers • Goal: Make sure we implement the right features • Ideally editable by a non-programmer • Tests features, not modules • Make sure you're done
  8. 8. Dagfinn Reiersøl, ABC Startsiden 8 Other functional tests • Goal: Make sure the implementation works • Could be testing the same thing as acceptance tests, but • need not be business-readable
  9. 9. Dagfinn Reiersøl, ABC Startsiden 9 The Test Double pattern
  10. 10. Dagfinn Reiersøl, ABC Startsiden 10 Surrounded by “Test Doubles”
  11. 11. Dagfinn Reiersøl, ABC Startsiden 11 Web testing: advantages • You can test exactly what the user sees • There are tools to test JavaScript-heavy / Ajax apps • There are tools to record and play back tests • “It seemed like an good idea at the time”
  12. 12. Dagfinn Reiersøl, ABC Startsiden 12 Web testing: disadvantages • Fragile: Small UI changes break tests • Fragile: External services break tests • Slow • Harder to replace external services with fakes
  13. 13. Dagfinn Reiersøl, ABC Startsiden 13 Strategies for robustness • Test Bus • Test patterns • Test refactoring • Making tests unnecessary
  14. 14. Dagfinn Reiersøl, ABC Startsiden 14 Web testing: essentials • Must be in a programming language (conditionals, variables) • Language not necessarily PHP, since HTTP allows communication between languages • “Business-readable” programming language (FitNesse wiki, Gherkin...)
  15. 15. Dagfinn Reiersøl, ABC Startsiden 15 Web testing tips • Test the HTML output using regex, XPath, CSS selector • Use element IDs or names to test links, forms and fields • Log HTTP requests in the application
  16. 16. Dagfinn Reiersøl, ABC Startsiden 16 How to use the request log Automated test fails but not manual run from browser • Run the automated test, find the failing request in the log. • Run the app manually, find the successful request in the log. • Compare.
  17. 17. Dagfinn Reiersøl, ABC Startsiden 17 Using SimpleTest <?php require_once 'simpletest/autorun.php'; require_once 'simpletest/web_tester.php'; error_reporting (E_ALL); class JoomlaSimpleTestTest extends WebTestCase { function testJoomla() { $this->get('http://localhost/joomla'); $this->assertResponse(200); $this->assertTitle('Welcome to the Frontpage'); $this->assertText('Welcome to the Frontpage'); $this->click('Welcome to Joomla!'); $this->assertPattern('/Joomla! is a free.*content publishing system/is'); } }
  18. 18. Dagfinn Reiersøl, ABC Startsiden 18 Using PHPUnit with Selenium <?php require_once 'PHPUnit/Extensions/SeleniumTestCase.php'; class JoomlaPhpUnitTest extends PHPUnit_Extensions_SeleniumTestCase { protected function setUp(){ $this->setBrowser('*firefox'); $this->setBrowserUrl('http://localhost/'); } public function testJoomla() { $this->open('http://localhost/joomla/'); $this->assertTitle('Welcome to the Frontpage'); $this->assertTextPresent('Welcome to the Frontpage'); $this ->clickAndWait( '//ul[@class='latestnews']/li[4]/a'); $this ->assertTextPresent('Joomla! is a free open'); } }
  19. 19. Dagfinn Reiersøl, ABC Startsiden 19 SimpleTest vs. PHPUnit / Selenium Simpletest: – Faster – Easier to use PHPUnit with Selenium – Heavier but more advanced – Can test JavaScript-heavy / Ajax apps – Can test cross-browser compatibility
  20. 20. Dagfinn Reiersøl, ABC Startsiden 20 Don't overdo Selenium “He ate a lot of brazil nuts which is a big deal because they contain selenium, which in high doses causes fatigue, vomiting, skin irritation, discharge from the fingernail beds and hair loss.” - House, M.D.
  21. 21. Dagfinn Reiersøl, ABC Startsiden 21 Acceptance test frameworks • FitNesse: has specific support for PHP • Cucumber: has PHP-related documentation • Robot Framework • Concordion
  22. 22. Dagfinn Reiersøl, ABC Startsiden 22 Using FitNesse with SimpleTest
  23. 23. Dagfinn Reiersøl, ABC Startsiden 23 PhpSlim Fixture for FitNesse test (simplistic!) <?php require_once 'simpletest/browser.php'; class SimpleTestFixture { function __construct() { $this->browser = new SimpleBrowser; } function goToPage($url) { $this->content = $this->browser->get($url); return !empty($this->content); } function pattern($pattern) { return preg_match("/$pattern/is",$this->content) > 0; } }
  24. 24. Dagfinn Reiersøl, ABC Startsiden 24 Using Cucumber with Webrat • joomla.feature (Cucumber Gherkin business-readable) • webrat_steps (Ruby code testing via HTTP) Feature: Joomla welcome In order to understand what Joomla is As a potential user I want to be able to read an introduction to Joomla Scenario: Welcome page Given I am on the home page When I follow "Welcome to Joomla!" Then I should see "Joomla! is a"
  25. 25. Dagfinn Reiersøl, ABC Startsiden 25 Webrat steps for Cucumber • Ruby code runs Webrat which runs the web app using HTTP Given /^I am on (.+)$/ do |page_name|   visit path_to(page_name) end When /^I follow "([^"]*)"$/ do |link|   click_link (link) end  Then /^I should see "([^"]*)"$/ do |text|   response_body.should contain(text) end
  26. 26. Dagfinn Reiersøl, ABC Startsiden 26 Avoiding difficult tests • It's hard to test behaviors that cut across the HTTP request • Improve architecture so these behaviors can't fail • Example: duplication between input control name and HTTP variable name
  27. 27. Dagfinn Reiersøl, ABC Startsiden 27 Hard to test: name duplication <input type="text" name="address1" value="<?= $address1 ?>"> // Oops... echo $_POST['address_1'];
  28. 28. Dagfinn Reiersøl, ABC Startsiden 28 Getting rid of name duplication • Use a form handling package • Or do it yourself. Simplistic version here: <?php define('ADDRESS1_NAME', 'address1'); $address1_value = $_POST[ADDRESS1_NAME]; <html> ... <input name="<?= ADDRESS1_NAME ?>" value="<?= $address1_value ?>">
  29. 29. Dagfinn Reiersøl, ABC Startsiden 29 Refactoring tests • Refactor or be doomed to drown in duplicate code • Especially true of web tests • Most important: eliminate duplication
  30. 30. Dagfinn Reiersøl, ABC Startsiden 30 BDD-style test names class JoomlaSimpleTestTest extends WebTestCase {     function setUp() {         $this­>get('http://localhost/joomla');     }     function testFrontPageIsValid() {         $this­>assertResponse(200);         $this­>assertTitle('Welcome to the Frontpage');         $this­>assertText('Welcome to the Frontpage');     }     function testWelcomePageIsValid() {         $this­>click('Welcome to Joomla!');         $this­>assertPattern('/Joomla! is a free.*content  publishing system/is');     } }
  31. 31. Dagfinn Reiersøl, ABC Startsiden 31 Using custom assertions instead class JoomlaSimpleTestTest extends WebTestCase {     function testJoomla() {         $this­>get('http://localhost/joomla');         $this­>assertValidHomePage();         $this­>click('Welcome to Joomla!');         $this­>assertValidWelcomePage();     }     function assertValidHomePage() {         $msg = 'Error on home page';         $this­>assertResponse(200,$msg);         $this­>assertTitle('Welcome to the Frontpage',$msg);         $this­>assertText('Welcome to the Fontpage',$msg);     }     function assertValidWelcomePage() {         $this­>assertPattern(            '/Joomla! is a free.*content publishing system/is',            'Error on welcome page');     }
  32. 32. Dagfinn Reiersøl, ABC Startsiden 32 ComposedRegex http://martinfowler.com/bliki/ComposedRegex.html • This regex tolerates markup (or anything else) between the content items • Yes, I know the email regex is simplistic $email = '.*w+@w+.w+.*';         $firstname = '.*w.*';         $lastname = '.*w.*';         $date = '.*20dd­dd­dd( |&nbsp;|T)dd:dd.*';         $regex = 'Email address:'. $email .                   'Name: ' . $firstname . $lastname .                   'Date:' . $date;
  33. 33. Dagfinn Reiersøl, ABC Startsiden 33 Re-using setup • Refactor gradually as needed: – When test class gets big, split it and... – ...extract setup method into a separate class – When setup is common to many tests... – ...extract setup class into a separate file class JoomlaTestSetup {     function setUp() {         $this­>get('http://localhost/joomla');     } } class JoomlaSimpleTestTest extends JoomlaTestSetup..
  34. 34. Dagfinn Reiersøl, ABC Startsiden 34 Useful types of classes in web testing • Scraper • Domain Object • Specification
  35. 35. Dagfinn Reiersøl, ABC Startsiden 35 Domain object test version class UserTest extends WebTestCase {     function setUp() {         $this­>user = new TestUser(             'Jane','Doe','jane@example.com');     }     function testCanRegisterUser...         $this­>setFieldByName(             'firstname',            $this­>user­>firstname         );
  36. 36. Dagfinn Reiersøl, ABC Startsiden 36 Given-when-then in PHP class JoomlaSimpleTestTest extends WebTestCase {     function setUp() {         $this­>pages = array(             'JoomlaHomePage' => 'http://localhost/joomla');     }     function testWelcomPageIsValid() {         $this­>givenIAmOn('JoomlaHomePage');         $this­>whenIFollow('Welcome to Joomla!');         $this­>thenIShouldSee('Joomla! is a free');     }     function givenIAmOn($name) {         $this­>get($this­>pages[$name]);     }     function whenIFollow($text) {        $this­>clickLink($text);      }     function thenIShouldSee($pattern) {         $this­>assertPattern("/$pattern/is");     }
  37. 37. Dagfinn Reiersøl, ABC Startsiden 37 Test bus http://www.martinfowler.com/ieeeSoftware/testBus.pdf
  38. 38. Dagfinn Reiersøl, ABC Startsiden 38 Recommendations from Uncle Bob • Test the features through the test bus • Test the user interface separately in isolation • Run a few end-to-end tests to test the “plumbing”
  39. 39. Dagfinn Reiersøl, ABC Startsiden 39 Test bus Where is the test bus? Some possbilities: • Test the Model • Test the Controller • Test the server side of an Ajax or “thin server” application • Test a public API
  40. 40. Dagfinn Reiersøl, ABC Startsiden 40 Test bus with presentation tests
  41. 41. Dagfinn Reiersøl, ABC Startsiden 41 Fake web service • Easier to do when not testing via HTTP • Sensing/verification is very hard to do
  42. 42. Dagfinn Reiersøl, ABC Startsiden 42 Self-initializing fake http://martinfowler.com/bliki/SelfInitializingFake.html
  43. 43. Dagfinn Reiersøl, ABC Startsiden 43 Self-initializing fake http://martinfowler.com/bliki/SelfInitializingFake.html
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×