Testing with PHPUnit and Selenium

Welcome! Testing with PHPUnit and Selenium Sebastian Bergmann http://sebastian-bergmann.de/ September 12 th 2007

Who I am

Sebastian Bergmann

Computer Scientist

Involved in the PHP Project since 2000

Author, Consultant, Coach, Trainer

Developer for eZ Systems AS

Who are you?

Your experience with

PHP 5?

OOP?

Testing?

PHPUnit?

Selenium?

Software Metrics?

Why test?

Companies develop more and more enterprise-critical applications with PHP.

Tests help to make sure that these applications work correctly.

Debugging Sucks, Testing Rocks! Debugging Sucks! Testing Rocks!

What needs testing?

Web Application

Backend

Business Logic

Reusable Components

Frontend

Form Processing, Templates, ...

“Rich Interfaces” with AJAX, JSON, ...

Feeds, Web Services, ...

And how do we test it?

Web Application

Backend

Functional Testing of the business logic with Unit Tests

Reusable Components

External components should have their own unit tests.

Frontend

Acceptance Tests or System Tests that are run “in the browser”

Testing of feeds, web services, etc. with unit tests

Compatibility Tests for Operating System / Browser combinations

Performance Tests and Security Tests

Testing Software

Component Tests

Executable code fragments, so called Unit Tests , test the correctness of parts, units , of the software ( system under test )

System Tests

Conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. (Wikipedia)

Non-Functional Tests

Performance, Stability, Security, ...

Testing Software

Developer Tests

Ensure that the code works correctly

Acceptance Tests

Ensure that the code does what the customer wants

How do you test code? <?php class Calculator { public function add ( $a , $b ) { return $a + $b ; } } ?>

How do you test code? <?php $fixture = new Calculator ; print $fixture -> add ( 0 , 1 ); $fixture = new Calculator ; print $fixture -> add ( 1 , 0 ); ?> 1 1

The tests are not specified in the code.

The tests are not automatically evaluated.

How do you test code?

The tests are specified in the code.

The tests are automatically evaluated.

The writing of tests is tedious.

<?php $fixture = new Calculator ; print $fixture -> add ( 0 , 1 ) == 1 ? 'pass' : 'fail' ; $fixture = new Calculator ; print $fixture -> add ( 1 , 0 ) == 1 ? 'pass' : 'fail' ; ?> pass pass

How do you test code?

The tests are automatically evaluated.

assertTrue() makes the writing of tests easy.

The test environment is not reusable.

<?php $fixture = new Calculator ; assertTrue ( $fixture -> add ( 0 , 1 ) == 1 ); $fixture = new Calculator ; assertTrue ( $fixture -> add ( 1 , 0 ) == 1 ); function assertTrue ( $expression ) { if (! $expression ) { throw new Exception ( 'Assertion failed.' ); } } ?>

Test Tools

To make (code) testing viable, good tool support is needed.

This is where a testing framework such as PHPUnit comes into play.

Requirements

Reusable test environment

Strict separation of production code and test code

Automatic execution of test code

Analysis of the result

Easy to learn to use and easy to use

Test Tools for PHP / Web

Unit Tests

PHPUnit

SimpleTest

System Tests

Selenium

PHPUnit + Selenium

Non-Functional Tests

Performance, Load, Stress, Availability, ...

ab, httperf, JMeter, Grinder, OpenSTA, ...

Security

Chorizo

PHPUnit: Webseite http://www.phpunit.de/

PHPUnit: Installation [email_address] ~ % pear channel-discover pear.phpunit.de Adding Channel "pear.phpunit.de" succeeded Discovery of channel "pear.phpunit.de" succeeded [email_address] ~ % pear install phpunit/phpunit Did not download optional dependencies: pear/Log, use --alldeps to download automatically phpunit/PHPUnit can optionally use package "pear/Log" downloading PHPUnit-3.1.8.tgz ... Starting to download PHPUnit-3.1.8.tgz (116,960 bytes) .........................done: 116,960 bytes install ok: channel://pear.phpunit.de/PHPUnit-3.1.8

PHPUnit: TextUI Testrunner [email_address] ~ % phpunit --help PHPUnit 3.2.0 by Sebastian Bergmann. Usage: phpunit [switches] UnitTest [UnitTest.php] . . .

Writing tests for PHPUnit <?php require_once 'PHPUnit/Framework.php' ; require_once 'Calculator.php' ; class CalculatorTest extends PHPUnit_Framework_TestCase { public function testAdd () { $calculator = new Calculator ; $this -> assertEquals ( 0 , $calculator -> add ( 0 , 0 )); $this -> assertEquals ( 1 , $calculator -> add ( 0 , 1 )); $this -> assertEquals ( 1 , $calculator -> add ( 1 , 0 )); $this -> assertEquals ( 2 , $calculator -> add ( 1 , 1 )); $this -> assertNotEquals ( 3 , $calculator -> add ( 1 , 2 )); } } ?>

Running tests with PHPUnit [email_address] ~ % phpunit CalculatorTest PHPUnit 3.2.0 by Sebastian Bergmann. F Time: 0 seconds There was 1 failure: 1) testAdd(CalculatorTest) Failed asserting that <integer:3> is not equal to <integer:3>. /home/sb/CalculatorTest.php:15 FAILURES! Tests: 1, Failures: 1.

Test Fixture

One of the most time-consuming parts of writing tests is writing the code to set the world up in a known state and then return it to its original state when the test is complete.

This known state is called the fixture of the test.

Test Fixture

In our CalculatorTest example the test fixture was a single object.

Most of the time, though, the fixture will be more complex than a simple object, and the amount of code needed to set it up will grow accordingly.

The actual content of the test gets lost in the noise of setting up the fixture.

This problem gets even worse when you write several tests with similar fixtures.

PHPUnit supports sharing the setup code through the setUp() and tearDown() template methods.

Writing tests for PHPUnit <?php require_once 'PHPUnit/Framework.php' ; require_once 'Calculator.php' ; class CalculatorTest extends PHPUnit_Framework_TestCase { protected $calculator ; protected function setUp () { $this -> calculator = new Calculator ; } public function testAdd () { $this -> assertEquals ( 0 , $this -> calculator -> add ( 0 , 0 )); } // ... ?>

Running tests with PHPUnit [email_address] ~ % phpunit CalculatorTest PHPUnit 3.2.0 by Sebastian Bergmann. ....F Time: 0 seconds There was 1 failure: 1) testAdd5(CalculatorTest) Failed asserting that <integer:3> is not equal to <integer:3>. /home/sb/CalculatorTest.php:26 FAILURES! Tests: 5, Failures: 1.

Philosophy

Without Tests

The developer thinks „the feature works“ ⇒ it works

With Tests

No tests for the feature ⇒ it does not work

PHPUnit API

For most use cases, PHPUnit has a very simple API

Extend PHPUnit_Framework_TestCase for your test case classes and

call assertTrue() or assertEquals() in your test methods

PHPUnit API

PHPUnit_Framework_Assert

Collection of assertion methods

PHPUnit_Framework_Test

Interface for all test objects

PHPUnit_Framework_TestCase

Abstract base class for test case classes

Child classes declare arbitrary number of tests

Template methods setUp() and tearDown()

PHPUnit API

PHPUnit_Framework_TestSuite

Collection of tests

Contains PHPUnit_Framework_Test objects, for instance PHPUnit_Framework_TestCase and PHPUnit_Framework_TestSuite objects

PHPUnit_Framework_TestResult

Collects test result information

Organizing Test Suites

Application/

Package/

Class (Application/Package/Class.php)

...

...

Tests/

AllTests.php

Package/

AllTests.php

ClassTest (Application/Tests/Package/ClassTest.php)

Organizing Test Suites <?php require_once 'PHPUnit/Framework.php' ; require_once 'Application/Tests/Package/AllTests.php' ; class AllTests { public static function suite () { $suite = new PHPUnit_Framework_TestSuite ( 'Project' ); $suite -> addTest ( Package_AllTests :: suite ()); return $suite ; } } ?>

Organizing Test Suites <?php require_once 'PHPUnit/Framework.php' ; require_once 'Application/Tests/Package/ClassTest.php' ; class Package_AllTests { public static function suite () { $suite = new PHPUnit_Framework_TestSuite ( 'Package' ); $suite -> addTestSuite ( 'Package_ClassTest' ); return $suite ; } } ?>

Logging XML <?xml version= "1.0" encoding= "UTF-8 " ?> <testsuites> <testsuite name= "CalculatorTest" file= "/home/sb/CalculatorTest.php" tests= "5" failures= "1" errors= "0" time= "0.0021438598632812" > <testcase name= "testAdd" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.00045299530029297" /> <testcase name= "testAdd2" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.0003299713134765" /> <testcase name= "testAdd3" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.0003290176391601" /> <testcase name= "testAdd4" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.0003290176391601" /> <testcase name= "testAdd5" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.00070285797119141" > <failure type= "PHPUnit_Framework_ExpectationFailedException" > Failed asserting that <integer:3> is not equal to <integer:3>. /home/sb/CalculatorTest.php:39 </failure> </testcase> </testsuite> </testsuites>

Logging JavaScript Object Notation (JSON) {"event":"suiteStart","suite":"CalculatorTest","tests":5} {"event":"test","suite":"CalculatorTest","test":"testAdd(CalculatorTest)", "status":"pass","time":0.0004529953,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd2(CalculatorTest)", "status":"pass","time":0.000329971313,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd3(CalculatorTest)", "status":"pass","time":0.000329017639,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd4(CalculatorTest)", "status":"pass","time":0.000329017639,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd5(CalculatorTest)", "status":"fail","time":0.000702857971,"trace":[],"message":"Failed asserting that <integer:3> is not equal to <integer:3>."}

Logging Test Anything Protocol (TAP) # TestSuite "CalculatorTest" started. ok 1 - testAdd(CalculatorTest) ok 2 - testAdd2(CalculatorTest) ok 3 - testAdd3(CalculatorTest) ok 4 - testAdd4(CalculatorTest) not ok 5 - Failure: testAdd5(CalculatorTest) # TestSuite "CalculatorTest" ended. 1..5

Logging GraphViz

Logging Software Metrics and Project Mess Detector

A software metric is a measure of some property of a piece of software or its specifications

Cyclomatic Complexity

NPath Complexity

Change Risk Analysis and Predictions (CRAP)

...

Violations of rules that are based on software metrics

Software Development Process

Unit Tests

play an essential role in several software development practices and processes

Test-First Programming

Test-Driven Development

Extreme Programming

allow for using the Design-by-Contract approach in programming languages that do not support it through language constructs

Contract that is to be followed when invoking a method

pre-condition, post-condition

semantic safety in addition to type safety

This slide contains material by Brady Kelly. Software Development Process Test-Driven Development

Method of designing software, not just a method of testing software

Test

What do we want X to do?

How do we want to tell X to do it?

How will we know when X has done it?

Code

How does X do it?

Tests drive the development

Tests written before code

No code without tests

Test-Driven Development The bank account example

Let's implement a class that represents a bank account using TDD

API

Retrieve the current balance

Set the balance

Deposit money

Withdraw money

Rules

Balance is initially zero

Balance cannot become negative

Test-Driven Development The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { private $ba ; protected function setUp () { $this -> ba = new BankAccount ; } public function testBalanceIsInitiallyZero () { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); } } ?>

Test-Driven Development The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { // ... public function testBalanceCannotBecomeNegative () { try { $this -> ba -> withdrawMoney ( 1 ); } catch ( BankAccountException $e ) { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); return; } $this -> fail (); } } ?>

Test-Driven Development The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { // ... public function testBalanceCannotBecomeNegative2 () { try { $this -> ba -> depositMoney (- 1 ); } catch ( BankAccountException $e ) { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); return; } $this -> fail (); } } ?>

Test-Driven Development The bank account example <?php require_once 'BankAccountException.php' ; class BankAccount { protected $balance = 0 ; public function getBalance () { return $this -> balance ; } protected function setBalance ( $balance ) { if ( $balance >= 0 ) { $this -> balance = $balance ; } else { throw new BankAccountException ; } } } ?>

Test-Driven Development The bank account example <?php require_once 'BankAccountException.php' ; class BankAccount { // ... public function depositMoney ( $amount ) { $this -> setBalance ( $this -> getBalance () + $amount ); return $this -> getBalance (); } public function withdrawMoney ( $amount ) { $this -> setBalance ( $this -> getBalance () - $amount ); return $this -> getBalance (); } } ?>

Test-Driven Development The bank account example [email_address] ~ % phpunit BankAccountTest PHPUnit 3.2.0 by Sebastian Bergmann. ... Time: 0 seconds OK (3 tests) [email_address] ~ % phpunit --testdox BankAccountTest PHPUnit 3.2.0 by Sebastian Bergmann. BankAccount - Balance is initially zero - Balance cannot become negative

Code Coverage

Via Xdebug, PHPUnit knows which LOC of the SUT are executed when a test runs

This information tells us, among other things, to what degree the SUT is covered by tests

100% code coverage is a required but not a sufficient criteria for test completeness

Code Coverage The bank account example [email_address] ~ % phpunit --report BankAccount BankAccountTest PHPUnit 3.2.0 by Sebastian Bergmann. ... Time: 0 seconds OK (3 tests) Generating report, this may take a moment.

Code Coverage The bank account example

Code Coverage The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { // ... public function testDepositWithdrawMoney () { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); $this -> ba -> depositMoney ( 1 ); $this -> assertEquals ( 1 , $this -> ba -> getBalance ()); $this -> ba -> withdrawMoney ( 1 ); $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); } } ?>

Code Coverage The bank account example

Microsoft Case Study

TDD project had twice the code quality

Writing the tests had a 15% productivity impact

IBM Case Study

40% less bugs

No productivity impact

John Deere / Ericsson Case Study

TDD lead to better code quality

Writing the tests had a 16% productivity impact

The other teams did not test at all

Software Development Process Test-Driven Development

Software Development Process Extreme Programming

The Test-First approach is part of Extreme Programming

Other aspects of Extreme Programming

Integration of the Customer

Customer sets iteration goals

Continous Integration

Pair Programming

Refactoring

Continous Integration

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily, leading to multiple integrations per day.

Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.

Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

Continous Integration D E M O CruiseControl

PHPUnit

Test Framework

Member of the xUnit family of test frameworks

Mock Objects

Integration

Selenium RC

Bamboo, Bitten, CruiseControl, ...

Even More Cool Stuff :-)

Code Coverage and Software Metrics

Test Database

Mutation Testing (GSoC07 project)

Selenium

Selenium

Test web applications in a web browser

Browser Compatibility Testing

System Functional Testing

Runs in the browser

Selenium IDE

IDE for Selenium tests

Extension for Firefox

Record, execute, edit, debug tests in the browser

Selenium

Selenium RC

Automated execution of Selenium tests

Tests can be specified in any language

PHP Bindings: PEAR Testing_Selenium

PHPUnit natively speaks the Selenium RC protocol

One test can be executed on multiple OS / Browser combinations

Selenium RC

Selenium Selenium RC

Selenium D E M O Selenium RC

Bibliography

T. Bhat, N. Nagappan, “Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies”, ISESE'06, September 21–22, 2006, Rio de Janeiro, Brazil.

L. Williams, E. M. Maximilien, and M. Vouk, “Test-Driven Development as a Defect-Reduction Practice”, Proceedings of IEEE International Symposium on Software Reliability Engineering, Denver, CO, pp. 34-45, 2003.