Testing Steve Loughran HP Laboratories June 9th, 2008

About me I break things for a living this is because I am clumsy breaking other people’s code is fun sometimes it is really hard

I break things for a living

this is because I am clumsy

breaking other people’s code is fun

sometimes it is really hard

Ant in Action the problems of complex builds in big teams not for this course stay in the IDE testing is key to getting big projects out the door automation comes next

the problems of complex builds in big teams

not for this course

stay in the IDE

testing is key to getting big projects out the door

automation comes next

your code doesn't work

I know this because...

my code doesn't work

it's OK to write code that doesn’t work

but not to ship it

especially when it matters

how do you get from "broken" to "shipping"?

Device Drivers ship and see test it! Test-Driven  x:  y : f ( y , x) > 0 Formal Methods prove it works

proofs of correctness shows an algorithm really works good for concurrency and other things you can’t test makes assumptions about system needs lots of maths/CS skills (Pi-Calculus, The HOL System) how do you do regression proofs?

shows an algorithm really works

good for concurrency

and other things you can’t test

makes assumptions about system

needs lots of maths/CS skills (Pi-Calculus, The HOL System)

how do you do regression proofs?

Testing: code that breaks unit test: test classes, libraries functional test: test the application system test: test a deployed system performance test: simulate load. acceptance tests: is it ready for use? a good test breaks the build!

unit test: test classes, libraries

functional test: test the application

system test: test a deployed system

performance test: simulate load.

acceptance tests: is it ready for use?

a good test breaks the build!

Test Driven Development any feature without a test doesn’t exist

Lots of unit test frameworks Java: JUnit, TestNG Python: PyUnit Ruby: Test:: Unit .NET: NUnit PHP: PHPUnit C++ (CppUnit)

Java: JUnit, TestNG

Python: PyUnit

Ruby: Test:: Unit

.NET: NUnit

PHP: PHPUnit

C++ (CppUnit)

JUnit main unit test framework in Java (see also TestNG) simple to learn good tool support somewhat low-level extensions for system testing

main unit test framework in Java (see also TestNG)

simple to learn

good tool support

somewhat low-level

extensions for system testing

public class LessSimpleTest extends TestCase { public LessSimpleTest (String s) { super(s); } public void testAssignment() { final Date date = new Date(); Event event = new Event(UUID.randomUUID(), date, "now", "Text"); assertEquals("self equality failed", event, event); assertEquals(date, event.getDate()); assertNotNull(date.getID()); } }

swing GUI TestRunner

Ant-generated report scales better; reports published

Continuous Integration see: CruiseControl, Hudson, LuntBuild

==> blame assignment whose change broke the build?

System Testing deploy, then: HtmlUnit, XmlUnit, Cactus, Selenium needs automated deployment, database setup, teardown performance testing by comparing timings w.r.t. old runs simulate entire behaviour of the client user/application

deploy, then: HtmlUnit, XmlUnit, Cactus, Selenium

needs automated deployment, database setup, teardown

performance testing by comparing timings w.r.t. old runs

simulate entire behaviour of the client user/application

Distributed Systems testing is still a research topic set up complex configurations run tests on remote machines collect results and log data post-mortem analysis virtualization can help see http://tinyurl.com/y99tez

testing is still a research topic

set up complex configurations

run tests on remote machines

collect results and log data

post-mortem analysis

virtualization can help

see http://tinyurl.com/y99tez

limitations of testing good tests are hard to write distributed tests very hard different system configurations are still a problem status of "tester" below "coder" A good test breaks the application

good tests are hard to write

distributed tests very hard

different system configurations are still a problem

status of "tester" below "coder"

A good test breaks the application

politics: resistance to change

developers: ignorance, fear E.E.E. Educate, Evangelise, Encourage gently show benefits use testing to track down a bug retain test for regression testing add tests for new code and old problems, not existing codebase adopt cruise control, reporting

E.E.E. Educate, Evangelise, Encourage

gently show benefits

use testing to track down a bug

retain test for regression testing

add tests for new code and old problems, not existing codebase

adopt cruise control, reporting

management: ignorance "too much time spent on testing" out of date with modern processes belief that testing slows the schedule (it does on the ship-and-see process) E.E.E., then D.D.D: Distract, Dissemble, Defeat

"too much time spent on testing"

out of date with modern processes

belief that testing slows the schedule (it does on the ship-and-see process)

E.E.E., then D.D.D: Distract, Dissemble, Defeat

call to action write tests use a CI tool: Hudson, Luntbuild, Cruise Control

write tests

use a CI tool: Hudson, Luntbuild, Cruise Control