Testing with PHPUnit and Selenium

Welcome! Testing with PHPUnit and Selenium Sebastian Bergmann http://sebastian-bergmann.de/ September 12 th 2007

Who I am
Sebastian Bergmann
Computer Scientist
Involved in the PHP Project since 2000
Author, Consultant, Coach, Trainer
Developer for eZ Systems AS

Who are you?
Your experience with
PHP 5?
OOP?
Testing?
PHPUnit?
Selenium?
Software Metrics?

Why test?
Companies develop more and more enterprise-critical applications with PHP.
Tests help to make sure that these applications work correctly.

Debugging Sucks, Testing Rocks! Debugging Sucks! Testing Rocks!

What needs testing?
Web Application
Backend
Business Logic
Reusable Components
Frontend
Form Processing, Templates, ...
“Rich Interfaces” with AJAX, JSON, ...
Feeds, Web Services, ...

And how do we test it?
Web Application
Backend
Functional Testing of the business logic with Unit Tests
Reusable Components
External components should have their own unit tests.
Frontend
Acceptance Tests or System Tests that are run “in the browser”
Testing of feeds, web services, etc. with unit tests
Compatibility Tests for Operating System / Browser combinations
Performance Tests and Security Tests

Testing Software
Component Tests
Executable code fragments, so called Unit Tests , test the correctness of parts, units , of the software ( system under test )
System Tests
Conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. (Wikipedia)
Non-Functional Tests
Performance, Stability, Security, ...

Testing Software
Developer Tests
Ensure that the code works correctly
Acceptance Tests
Ensure that the code does what the customer wants

How do you test code? <?php class Calculator { public function add ( $a , $b ) { return $a + $b ; } } ?>

How do you test code? <?php $fixture = new Calculator ; print $fixture -> add ( 0 , 1 ); $fixture = new Calculator ; print $fixture -> add ( 1 , 0 ); ?> 1 1
The tests are not specified in the code.
The tests are not automatically evaluated.

How do you test code?
The tests are specified in the code.
The tests are automatically evaluated.
The writing of tests is tedious.
<?php $fixture = new Calculator ; print $fixture -> add ( 0 , 1 ) == 1 ? 'pass' : 'fail' ; $fixture = new Calculator ; print $fixture -> add ( 1 , 0 ) == 1 ? 'pass' : 'fail' ; ?> pass pass

How do you test code?
The tests are automatically evaluated.
assertTrue() makes the writing of tests easy.
The test environment is not reusable.
<?php $fixture = new Calculator ; assertTrue ( $fixture -> add ( 0 , 1 ) == 1 ); $fixture = new Calculator ; assertTrue ( $fixture -> add ( 1 , 0 ) == 1 ); function assertTrue ( $expression ) { if (! $expression ) { throw new Exception ( 'Assertion failed.' ); } } ?>

Test Tools
To make (code) testing viable, good tool support is needed.
This is where a testing framework such as PHPUnit comes into play.
Requirements
Reusable test environment
Strict separation of production code and test code
Automatic execution of test code
Analysis of the result
Easy to learn to use and easy to use

Test Tools for PHP / Web
Unit Tests
PHPUnit
SimpleTest
System Tests
Selenium
PHPUnit + Selenium
Non-Functional Tests
Performance, Load, Stress, Availability, ...
ab, httperf, JMeter, Grinder, OpenSTA, ...
Security
Chorizo

PHPUnit: Webseite http://www.phpunit.de/

PHPUnit: Installation [email_address] ~ % pear channel-discover pear.phpunit.de Adding Channel "pear.phpunit.de" succeeded Discovery of channel "pear.phpunit.de" succeeded [email_address] ~ % pear install phpunit/phpunit Did not download optional dependencies: pear/Log, use --alldeps to download automatically phpunit/PHPUnit can optionally use package "pear/Log" downloading PHPUnit-3.1.8.tgz ... Starting to download PHPUnit-3.1.8.tgz (116,960 bytes) .........................done: 116,960 bytes install ok: channel://pear.phpunit.de/PHPUnit-3.1.8

PHPUnit: TextUI Testrunner [email_address] ~ % phpunit --help PHPUnit 3.2.0 by Sebastian Bergmann. Usage: phpunit [switches] UnitTest [UnitTest.php] . . .

Writing tests for PHPUnit <?php require_once 'PHPUnit/Framework.php' ; require_once 'Calculator.php' ; class CalculatorTest extends PHPUnit_Framework_TestCase { public function testAdd () { $calculator = new Calculator ; $this -> assertEquals ( 0 , $calculator -> add ( 0 , 0 )); $this -> assertEquals ( 1 , $calculator -> add ( 0 , 1 )); $this -> assertEquals ( 1 , $calculator -> add ( 1 , 0 )); $this -> assertEquals ( 2 , $calculator -> add ( 1 , 1 )); $this -> assertNotEquals ( 3 , $calculator -> add ( 1 , 2 )); } } ?>

Running tests with PHPUnit [email_address] ~ % phpunit CalculatorTest PHPUnit 3.2.0 by Sebastian Bergmann. F Time: 0 seconds There was 1 failure: 1) testAdd(CalculatorTest) Failed asserting that <integer:3> is not equal to <integer:3>. /home/sb/CalculatorTest.php:15 FAILURES! Tests: 1, Failures: 1.

Test Fixture
One of the most time-consuming parts of writing tests is writing the code to set the world up in a known state and then return it to its original state when the test is complete.
This known state is called the fixture of the test.

Test Fixture
In our CalculatorTest example the test fixture was a single object.
Most of the time, though, the fixture will be more complex than a simple object, and the amount of code needed to set it up will grow accordingly.
The actual content of the test gets lost in the noise of setting up the fixture.
This problem gets even worse when you write several tests with similar fixtures.
PHPUnit supports sharing the setup code through the setUp() and tearDown() template methods.

Writing tests for PHPUnit <?php require_once 'PHPUnit/Framework.php' ; require_once 'Calculator.php' ; class CalculatorTest extends PHPUnit_Framework_TestCase { protected $calculator ; protected function setUp () { $this -> calculator = new Calculator ; } public function testAdd () { $this -> assertEquals ( 0 , $this -> calculator -> add ( 0 , 0 )); } // ... ?>

Running tests with PHPUnit [email_address] ~ % phpunit CalculatorTest PHPUnit 3.2.0 by Sebastian Bergmann. ....F Time: 0 seconds There was 1 failure: 1) testAdd5(CalculatorTest) Failed asserting that <integer:3> is not equal to <integer:3>. /home/sb/CalculatorTest.php:26 FAILURES! Tests: 5, Failures: 1.

Philosophy
Without Tests
The developer thinks „the feature works“ ⇒ it works
With Tests
No tests for the feature ⇒ it does not work

PHPUnit API
For most use cases, PHPUnit has a very simple API
Extend PHPUnit_Framework_TestCase for your test case classes and
call assertTrue() or assertEquals() in your test methods

PHPUnit API

PHPUnit API
PHPUnit_Framework_Assert
Collection of assertion methods
PHPUnit_Framework_Test
Interface for all test objects
PHPUnit_Framework_TestCase
Abstract base class for test case classes
Child classes declare arbitrary number of tests
Template methods setUp() and tearDown()

PHPUnit API
PHPUnit_Framework_TestSuite
Collection of tests
Contains PHPUnit_Framework_Test objects, for instance PHPUnit_Framework_TestCase and PHPUnit_Framework_TestSuite objects
PHPUnit_Framework_TestResult
Collects test result information

Organizing Test Suites
Application/
Package/
Class (Application/Package/Class.php)
...
...
Tests/
AllTests.php
Package/
AllTests.php
ClassTest (Application/Tests/Package/ClassTest.php)

Organizing Test Suites <?php require_once 'PHPUnit/Framework.php' ; require_once 'Application/Tests/Package/AllTests.php' ; class AllTests { public static function suite () { $suite = new PHPUnit_Framework_TestSuite ( 'Project' ); $suite -> addTest ( Package_AllTests :: suite ()); return $suite ; } } ?>

Organizing Test Suites <?php require_once 'PHPUnit/Framework.php' ; require_once 'Application/Tests/Package/ClassTest.php' ; class Package_AllTests { public static function suite () { $suite = new PHPUnit_Framework_TestSuite ( 'Package' ); $suite -> addTestSuite ( 'Package_ClassTest' ); return $suite ; } } ?>

Logging XML <?xml version= "1.0" encoding= "UTF-8 " ?> <testsuites> <testsuite name= "CalculatorTest" file= "/home/sb/CalculatorTest.php" tests= "5" failures= "1" errors= "0" time= "0.0021438598632812" > <testcase name= "testAdd" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.00045299530029297" /> <testcase name= "testAdd2" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.0003299713134765" /> <testcase name= "testAdd3" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.0003290176391601" /> <testcase name= "testAdd4" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.0003290176391601" /> <testcase name= "testAdd5" class= "CalculatorTest" file= "/home/sb/CalculatorTest.php" time= "0.00070285797119141" > <failure type= "PHPUnit_Framework_ExpectationFailedException" > Failed asserting that <integer:3> is not equal to <integer:3>. /home/sb/CalculatorTest.php:39 </failure> </testcase> </testsuite> </testsuites>

Logging JavaScript Object Notation (JSON) {"event":"suiteStart","suite":"CalculatorTest","tests":5} {"event":"test","suite":"CalculatorTest","test":"testAdd(CalculatorTest)", "status":"pass","time":0.0004529953,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd2(CalculatorTest)", "status":"pass","time":0.000329971313,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd3(CalculatorTest)", "status":"pass","time":0.000329017639,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd4(CalculatorTest)", "status":"pass","time":0.000329017639,"trace":[],"message":""} {"event":"test","suite":"CalculatorTest","test":"testAdd5(CalculatorTest)", "status":"fail","time":0.000702857971,"trace":[],"message":"Failed asserting that <integer:3> is not equal to <integer:3>."}

Logging Test Anything Protocol (TAP) # TestSuite "CalculatorTest" started. ok 1 - testAdd(CalculatorTest) ok 2 - testAdd2(CalculatorTest) ok 3 - testAdd3(CalculatorTest) ok 4 - testAdd4(CalculatorTest) not ok 5 - Failure: testAdd5(CalculatorTest) # TestSuite "CalculatorTest" ended. 1..5

Logging GraphViz

Logging Software Metrics and Project Mess Detector
A software metric is a measure of some property of a piece of software or its specifications
Cyclomatic Complexity
NPath Complexity
Change Risk Analysis and Predictions (CRAP)
...
Violations of rules that are based on software metrics

Software Development Process
Unit Tests
play an essential role in several software development practices and processes
Test-First Programming
Test-Driven Development
Extreme Programming
allow for using the Design-by-Contract approach in programming languages that do not support it through language constructs
Contract that is to be followed when invoking a method
pre-condition, post-condition
semantic safety in addition to type safety

This slide contains material by Brady Kelly. Software Development Process Test-Driven Development
Method of designing software, not just a method of testing software
Test
What do we want X to do?
How do we want to tell X to do it?
How will we know when X has done it?
Code
How does X do it?
Tests drive the development
Tests written before code
No code without tests

Test-Driven Development The bank account example
Let's implement a class that represents a bank account using TDD
API
Retrieve the current balance
Set the balance
Deposit money
Withdraw money
Rules
Balance is initially zero
Balance cannot become negative

Test-Driven Development The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { private $ba ; protected function setUp () { $this -> ba = new BankAccount ; } public function testBalanceIsInitiallyZero () { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); } } ?>

Test-Driven Development The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { // ... public function testBalanceCannotBecomeNegative () { try { $this -> ba -> withdrawMoney ( 1 ); } catch ( BankAccountException $e ) { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); return; } $this -> fail (); } } ?>

Test-Driven Development The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { // ... public function testBalanceCannotBecomeNegative2 () { try { $this -> ba -> depositMoney (- 1 ); } catch ( BankAccountException $e ) { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); return; } $this -> fail (); } } ?>

Test-Driven Development The bank account example <?php require_once 'BankAccountException.php' ; class BankAccount { protected $balance = 0 ; public function getBalance () { return $this -> balance ; } protected function setBalance ( $balance ) { if ( $balance >= 0 ) { $this -> balance = $balance ; } else { throw new BankAccountException ; } } } ?>

Test-Driven Development The bank account example <?php require_once 'BankAccountException.php' ; class BankAccount { // ... public function depositMoney ( $amount ) { $this -> setBalance ( $this -> getBalance () + $amount ); return $this -> getBalance (); } public function withdrawMoney ( $amount ) { $this -> setBalance ( $this -> getBalance () - $amount ); return $this -> getBalance (); } } ?>

Test-Driven Development The bank account example [email_address] ~ % phpunit BankAccountTest PHPUnit 3.2.0 by Sebastian Bergmann. ... Time: 0 seconds OK (3 tests) [email_address] ~ % phpunit --testdox BankAccountTest PHPUnit 3.2.0 by Sebastian Bergmann. BankAccount - Balance is initially zero - Balance cannot become negative

Code Coverage
Via Xdebug, PHPUnit knows which LOC of the SUT are executed when a test runs
This information tells us, among other things, to what degree the SUT is covered by tests
100% code coverage is a required but not a sufficient criteria for test completeness

Code Coverage The bank account example [email_address] ~ % phpunit --report BankAccount BankAccountTest PHPUnit 3.2.0 by Sebastian Bergmann. ... Time: 0 seconds OK (3 tests) Generating report, this may take a moment.

Code Coverage The bank account example

Code Coverage The bank account example <?php require_once 'PHPUnit/Framework.php' ; require_once 'BankAccount.php' ; class BankAccountTest extends PHPUnit_Framework_TestCase { // ... public function testDepositWithdrawMoney () { $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); $this -> ba -> depositMoney ( 1 ); $this -> assertEquals ( 1 , $this -> ba -> getBalance ()); $this -> ba -> withdrawMoney ( 1 ); $this -> assertEquals ( 0 , $this -> ba -> getBalance ()); } } ?>

Code Coverage The bank account example

Microsoft Case Study
TDD project had twice the code quality
Writing the tests had a 15% productivity impact
IBM Case Study
40% less bugs
No productivity impact
John Deere / Ericsson Case Study
TDD lead to better code quality
Writing the tests had a 16% productivity impact
The other teams did not test at all
Software Development Process Test-Driven Development

Software Development Process Extreme Programming
The Test-First approach is part of Extreme Programming
Other aspects of Extreme Programming
Integration of the Customer
Customer sets iteration goals
Continous Integration
Pair Programming
Refactoring

Continous Integration
Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily, leading to multiple integrations per day.
Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.
Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

Continous Integration D E M O CruiseControl

PHPUnit
Test Framework
Member of the xUnit family of test frameworks
Mock Objects
Integration
Selenium RC
Bamboo, Bitten, CruiseControl, ...
Even More Cool Stuff :-)
Code Coverage and Software Metrics
Test Database
Mutation Testing (GSoC07 project)

Selenium
Selenium
Test web applications in a web browser
Browser Compatibility Testing
System Functional Testing
Runs in the browser
Selenium IDE
IDE for Selenium tests
Extension for Firefox
Record, execute, edit, debug tests in the browser

Selenium
Selenium RC
Automated execution of Selenium tests
Tests can be specified in any language
PHP Bindings: PEAR Testing_Selenium
PHPUnit natively speaks the Selenium RC protocol
One test can be executed on multiple OS / Browser combinations
Selenium RC

Selenium Selenium RC

Selenium D E M O Selenium RC

Bibliography
T. Bhat, N. Nagappan, “Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies”, ISESE'06, September 21–22, 2006, Rio de Janeiro, Brazil.
L. Williams, E. M. Maximilien, and M. Vouk, “Test-Driven Development as a Defect-Reduction Practice”, Proceedings of IEEE International Symposium on Software Reliability Engineering, Denver, CO, pp. 34-45, 2003.
B. George and L. Williams, “A Structured Experiment of Test-Driven Development”, Information and Software Technology (IST), 46(5), pp. 337-342, 2003.

http://sebastian-bergmann.de	601
http://www.planet-php.net	29
http://www.techgig.com	14
http://www.slideshare.net	9
http://www.planet-php.org	7
http://planet-php.org	5
http://www.netvibes.com	3
http://www.phpeye.com	3
http://lj-toys.com	2
http://planet-php.net	1
http://localhost	1
http://s3.amazonaws.com	1

Testing with PHPUnit and Selenium

by Sebastian Bergmann on Sep 13, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

12 Embeds 676

Statistics

Testing with PHPUnit and Selenium — Presentation Transcript