Functional testing as a core pillar in Agile
Agile software development is based on a good number of best practices. Some best
practices are taken from previous methodologies and some are totally new. There’s at
least one best practice which is common to any software development project;
As an agile development group, our goal is to release stable versions of Plastic SCM at
a frequent pace. In order to accomplish this task, Códice relies on an automated
testing to heavily supplement our QA process. By utilizing automated test cases Códice
has vastly increased the breadth of the test plan without breaking the bank. The ever
increasing test case repository continues to expand which provides a more thorough
functional and regression testing cycle with each release.
Many best practices advocate automated testing but only at the code level and neglect
the behavioral aspects of an applications front end. Plastic SCM relies on a
combination of unit tests provided by NUnit and PNUnit (an extension to the base
NUnit for distributed testing we’ve implemented and released together with NUnit
2.5) that perform logic tests at the code layer. Our QA process also contains a large
suite of behavioral tests that provide automated functional and regression testing
from the external user perspective, via the graphical user interface, which is provided
by AutomatedQA’s TestComplete.
Our agile development practices utilize both unit and behavioral testing combinations
to ensure the greatest amount of testing coverage that in turn ensures exceptional
quality for Plastic SCM releases.
A development cycle based on agile principles
Before entering into the test process itself, it is required to understand how the Plastic
SCM development life cycle is defined.
Plastic SCM is a version control system. A high‐end system featuring a large number of
unique and best of breed versioning features. Plastic has been under heavy
development for the last three years and a half, which means it has experienced a
Figure 1. Plastic SCM code base evolution
Today Plastic has about 300K lines of code in 5 different programming languages. It is
multi‐platform, distributed and includes complex features like branching and multi‐
server data replication all sitting on top of a state of the art, graphical user interface.
High levels of complexity, rapidly expanding codebase and a continuously growing
development team has placed an immense amount of importance on the quality
Introducing changes at high speed basically means there’re a lot of opportunities to
break code. Every change has the potential risk of breaking code. The Plastic code base
has about 100K revisions, 4000 branches (please note that we use branch per task
pattern) and about 30K changesets.
There are two basic principles (which can be traced back to Rapid Application
Development ) behind the way Codice develops Plastic which are:
• Always be ready to release; be prepared to count on a fallback solution.
• Increase visibility; which means keep everyone inside and outside the project
(Pigs and Chickens as introduced in the prologue of the SCRUM book ).
A number of agile principles can be linked with the previous two; from release often to
take small steps and many others.
Enter the SCRUM
The entire Plastic SCM development process is managed using SCRUM. The
methodology is strict enough for a project with tight deadlines, and flexible enough to
manage agile practices.
Figure 2. SCRUM cycle
SCRUM is so versatile it is even compatible with CMMi. We achieved CMMi Level 2 one
year and a half ago, based on agile practices and a really narrow set of tools that
• A wiki for documentation.
• Our internal bug tracking system for requirements, new functionality, feature
requests and bugs.
• Plastic for version control. (We do eat our own dog’s food)
• And a combination of tools for testing:
o NUnit for unit testing.
o Parallel NUnit for multi‐platform command line testing (PNUnit).
o Test Complete for functional testing.
Our testing set up exceeded the CMMi Level 2 requirements. For us it was more
important to implement a process that adds more value to Plastic.
We run sprints with two to four weeks duration (it has been tuned cycle after cycle)
and we track progress and use special burndown charts on our internal tts.
Figure 3. A burndown chart in tts
We use SCRUM to focus on stability, and that’s what we try to achieve release after
The complete lifecycle
The Plastic development strategy can be summarized in the following picture.
Figure 4. Internal development cycle
Developers create and run tests while they develop code. We try to follow a test driven
development strategy, and we definitely follow a bug fixing test driven strategy which
means a regression is written prior to bug fixing, so we make sure it will never happen
again. We set up a separate quality assurance group but were not as happy with the
results compared to developer based automated testing, with the exception of
Functional testing user stories; closing the gap
User stories are central pieces to manage requirements and user communication in an
Testing methods based on xUnit family tools introduce a gap between what the user
sees and understands, and the internal program logic. A number of unit tests are
written today following the test driven development approach, which means testing is
meant to be a tool for developers to write more solid code. It can be used later for
regression testing but at its purest meaning TDD is used during development only. You
don’t write test cases to test for every possibility, as that’s simply unrealistic, but you
can cover development changes as they occur
Of course mastering Unit Testing helps building a test suite based project safety net.
The gap is that xUnit tools talk about code, while project stakeholders talk about
Functional testing using tools to automate the user interface testing specifically
address the gap covering the scenarios defined in close collaboration with the
customers. Both acceptance and regression can be covered.
As usual, there’s no one‐size‐fits‐all solution for testing; a combination of techniques
will create a better strategy than relying on a single testing method.
Testing for refactoring
One of the techniques glorified by agile methods is refactoring. Refactoring is not only
about beautiful and clever code, it’s about economics; you keep a better baseline and
your project/product will have an added value. You’ll be able to adapt faster to
changes (and agile is all about change!), and your code base will have a larger and
I like to compare the following two graphs showing the cost of change:
Traditionally, changes were thought to cost more as time passed by. With the rise of
agile, it seems there’s a way to keep costs under control.
So, in the end, like in all engineering disciplines, cost is an important driver. Refactoring
is not only about keeping nice code for developer’s joy but one of the key pieces to
keep costs under control.
That being said, it is important to note that refactoring can only exist thanks to testing.
Yes, we all learn about refactoring with a “Hello World” application and using our
favorite IDE, but in real life you can only change code when you have tests covering it.
Test coverage allows you to freely redesign or rewrite blocks of code you’re unhappy
with. If you’ve already been in the business for a long time, you’ve probably faced a
situation where poorly written code is kept unmodified simply because there’s no way
to guarantee it will still work after a major (or minor!) rewrite. No one wants to be
responsible for breaking it, so some systems have the rotten apple effect; the core,
which is supposed to be one of the strongest parts of the system, is totally broken.
Testing changes the whole picture and GUI testing is not a small part here; a priority
one bug can be related to an action the user interface can’t handle correctly. Maybe
there are thousands of tests covering the core that won’t break, but a simple mistake
on the UI layer can ruin the whole effort. So, again, the key is GUI test coverage.
Since day one we have been writing unit tests using NUnit. NUnit covers the unit
testing which is needed on a daily basis (for every build) but we also needed to extend
NUnit in order to create more complete testing scenarios.
We extended NUnit with our own modifications to create PNUnit (which has been fully
integrated into NUnit 2.5) and which stands for Parallel NUnit. Using PNUnit we’re able
to run tests on different machines that are able to synchronize with each other in
order to create more complex scenarios such us; clients and servers on different OSs
(different Linux distributions, Windows 2000, XP, Vista, Solaris...), or under different
But there are limitations to unit testing. With PNUnit alone we could not test the
graphical user interface nor could we test our plug‐ins to IDE’s such as Visual Studio
Thankfully we introduced TestComplete only a few weeks after the project was started
(at the same time PUNit was finished). TestComplete gave us high‐end functionality at
a fraction of the cost (some team members were used to high‐cost testing systems and
were initially concerned about learning a new environment).
TestComplete also provides additional testing coverage by simulating users interacting
with the graphical user interface. Tests can be data driven and iterative which means a
small number of test scripts can be used to perform a large number of operations
repeatedly. This is useful for running multiple scenarios’s using conditional expressions
and looking for performance bottlenecks and stress points. TestComplete (and more
specifically its runners) are used on a daily basis to run regression testing.
The unit test scripts and the TestComplete test scripts are run after each build and
complement each other which provide Plastic SCM with testing coverage inside and
Testing set up
In the beginning, we only used TestComplete runners to execute release tests. We
wrote a test suite which consists of a number of small test cases acting on the GUI and
We used the record functionality to test the Plastic 1.x series, and then moved to a
more code based approach when we moved the entire test suite to Plastic 2.x.
Creating tests is now much easier for developers (a different approach would be
needed if non technical people were required to create test suites, as normally
happens with business software). Of course it is important to note that a test suite is
never complete. It evolves as the software evolves and it changes to reflect
modifications in the product. It is an ongoing effort which pays off release after release
when it acts as a safety net being able to catch bugs before a new release is delivered
Once the test suite was in place, we started using it in combination with the runners to
test the software under certain conditions. We run the same, ever growing test suite,
release after release (which normally happens once a week) on Windows XP, 2000,
2003 and Vista and with different .NET frameworks. We take advantage of the
synchronization facilities of TestComplete to run separate servers and clients on
We make heavy use of VMWare and virtual machines installed with different operating
systems to leverage the limited number of testing machines available.
The test cases will relate back to a user story which then relates to individual tasks.
When a problem occurs, a developer can quickly locate the user story and the
individual tasks associated with the user story. From there it is a very short distance
from the task to the source code where the problem can be identified and corrected.
The branch by task methodology we use and our functional testing suite complement
each other to simplify the bug fixing process.
Once our test suite was in place, we employed it against our build and test systems.
Each time a developer finishes a test, his changes are uploaded onto a test machine,
and there the whole TestComplete test suite is run reporting potential errors on fail.
Only one single combination is run on task completion (Google search for “pairwise
testing”), and the whole test suite covering all the OS’s is run only when the new
baseline is created. This reduces the test time for daily and weekly builds and expands
the test coverage releases.
Figure 5. Testing set up
Today products have to evolve at a really rapid pace. Agile methods are a key factor,
but testing is one of the three core pillars (together with version control and defect
tracking) to build upon. There are very few situations, especially in product
development, where the manual testing effort can provide sufficient test coverage.
Automated functional testing allows describing scenarios, which can be traced back to
user stories, to be tested again and again, release after release. The combination of
functional and unit testing gives Plastic SCM a much greater testing scope then can be
achieved by manual testing or simply using one over the other. By using all three,
functional, unit and manual testing we can provide our customers with a stable
platform, a high degree of confidence in our company and our roadmap for success
that ensures the quality of Plastic SCM now and in the future as the company and the
codebase continue to grow at a rapid pace.
1. Rapid Application Development. Steve McConnell. Microsoft Press. 1996.
2. Agile Project Management with Scrum. Ken Schwaber. Microsoft Press. 2004.
3. Scrum Meets CMMi. Pablo Santos. Dr. Dobbs Magazine. August 2007.