Enabling Agile Testing Through
Continuous Integration
Sean Stolberg, Pacific Northwest National Laboratory

Where We’re Going Today
Introduction - Being thrown into the deep end of Agile Testing
Our Plan – Deciding to Implement Agile Practices
The Problem
The Solution - Continuous Integration (CI)
What is CI?
Implementing CI
Assessing the Team’s CI Practices
Assessing Our New Agile Development Practices and Testing
Capabilities
Retrospective

Being Thrown Into the Deep End
Me
 Eight years as traditional software tester/QA engineer.
 November 2007 – First job as a tester on an Agile team.
First two sprints
 Humbling
 All the “moves” I knew didn’t work too well.
 Kept coming up short on time, test coverage, and accumulation of
technical debt.
Re-evaluation
 “What I’m doing is not working, so what will?”

Practices We Decided to Implement
1. Define and execute “just-enough” acceptance tests. [1]
2. Automate as close to 100% of the acceptance tests as
possible. [2]
3. Automate acceptance tests using a “subcutaneous” test
approach with a xUnit test framework. [2]
[1] “eXtreme Rules of the Road: How a tester can steer an eXtreme Programming project toward success”, Lisa Crispin,
STQE Jul/Aug 2001
[2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley

Practices We Decided to Implement (continued)
5. Run all acceptance tests in the regression test suite with
the build, daily (at a minimum). [4]
6. Develop unit tests for all new code during a sprint. [5]
7. Run all unit tests with every build. [4]
8. Run multiple builds per day. [4]
[4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html
[5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html

The Problem
We didn’t have an automation framework.
How could we:
 Implement our new development practices?
 Run all acceptance tests in the regression test suite with the
build?
 Run all unit tests with every build?
 Run multiple builds per day?

The Solution
Continuous Integration
 Commonly used by teams already successfully implementing
Agile Testing.
 Provides framework to implement several of the practices.
 We needed to find out more.

What is Continuous Integration?
Describes a set of software engineering practices that
speed up the delivery of software
by
decreasing integration times
Emerged in the XP community.
Martin Fowler and Kent Beck – helped make it popular.

What CI Might Look Like
1. Automated source repository change detection
 Change is detected resulting to a chain of events being put
into motion
2. Get latest source
3. Compile (or run through other check; interpreted code)
4. Run unit tests
5. Deploy to test environment
6. Run acceptance tests
7. Send results of testing
8. Publish build if tests pass

Speeding Up the Feedback Loop
Compile
Report Unit Testing
Continuous
Integration
Notify Deploy
Acceptance
Publish build
Testing

Tools Used

Virtual Server Host
• First step in build
automation
• Configuration
• Dual Quad Core Xeon
1.6ghz
• 8GB Ram
• 4 x 146GB 15k RPM
SCSI drives in RAID-5.
• Unified build environment
and administration
• Get build machines out
from under the desks and
into the server room

Virtual Build Server
IF a source code change is detect
• Refresh source code
• Build (ABS using VS2008)
• Clean build
• Build All
• Build Installer
IF no build errors were found
• Prep the test VM for product
install and testing (VBScript
helper script called via STAF)
• Copy install files to test VM
(STAF)
• Install app-under-test on test
VM (STAF)
• Discover and deploy all Unit
and Acceptance tests to test
VM (custom C# helper
app/STAF)
• Execute tests on test VM
(custom C# helper app/STAF)
• Copy test result .xml file(s)
from test VM to build machine
(STAF)
• Process the test result .xml
files (custom C# helper app)

Communication
Send mail with
• test results (PASS or FAIL
with details of failures)
• Link to location of build
• Build logs
• Change logs

Assessing our CI Practices
Maintain a Single Source Repository
Before CI: Seapine Surround SCM source repository.
After CI: No change.
Automate the Build
Before CI: Some partial automation; still very manual.
Heterogeneous build environments.
After CI: Yes, with Automated Build Studio (ABS).
Make Your Build Self-Testing
Before CI: Not self testing.
After CI: NUnit framework unit and acceptance tests.

Assessing our CI Practices (continued)
Everyone Commits Every Day
Before CI: Unknown, varied probably.
After CI: We can only hope.
Every Commit Should Build the Mainline on an Integration Machine
Before CI: No, it was not happening.
After CI: Yes, ABS's Continuous Integration Tasks helped us do
this.
Keep the Build Fast
Before CI: Multi-pass builds, unordered dependencies.
After CI: 10 to 15 min; refactoring needed sooner than later.

Assessing our CI Practices (continued)
Test in a Clone of the Production Environment
Before CI: Yes, but not automated
After CI: Using clean virtual machine test clients to install and test
Make it Easy for Anyone to Get the Latest Executable
Before CI: Not all projects using the common build repository.
Some private file share locations for production code
After CI: All products building to common location now. Build mail
contains link to new build location

Assessing our CI Practices (continued)
Everyone can see what's happening
Before CI: Limited to ad hoc e-mails, no web, no reporting,
different project worked differently.
After CI: Use ABS's web interface to see the progress of builds,
and email build and test status.
Automate Deployment
Before CI: Not being done.
After CI: Yes, automatically deploy the build, then test it.

Assessing Our Agile Practices
Define and execute “just-enough” acceptance tests
 Made acceptance test definition a required task during sprint
planning.
 Developers and customer liked visibility into testing.
Automate as close to 100% of the acceptance tests as possible
 Now tried to automate as close to 100% acceptance tests as
possible.
 Slowed our accumulation of technical test debt.

Assessing Our Agile Practices (continued)
Automate acceptance tests using a “subcutaneous” test approach
with an xUnit test framework
 NUnit test framework used to automate our acceptance/functional
testing using the API of the application.
 Acceptance tests versioned code in the repository and possible to
run in a daily build.
Run all acceptance tests in the regression test suite with the build,
daily (at a minimum)
 NUnit acceptance tests now ran with a daily build.
 Practice enabled using our new continuous integration system.

Assessing Our Agile Practices (continued)
Develop unit tests for all new code during a sprint
 Developers started putting more emphasis on developing unit
tests during the same sprint.
 Practice encouraged due to ease with which unit tests could now
be run with the build.
Run all unit tests with every build
 NUnit tests now ran with every build.
Run multiple builds per day
 Builds started automatically when changes to the source
repository were checked in.

Retrospective
Personal “biggest discoveries”
 Testing in sync with development.
Embracing fully automated acceptance tests
 Exciting and intimidating.
Hard to convince developers to go through the automation “hump-of-pain”
 But it’s worth it.
 Devoting personal time toward investigation and proof-of-concept paid
off.
Can’t imagine working on a team not using Agile Testing or development
 Get “hooked” on injecting testing and quality early into the cycle.
 Adds immediate value.

Recommended
Reading
Crispin, Lisa, Tip House. Testing
Extreme Programming. Addison-
Wesley Professional, 2002
Gerard Meszaros, xUnit Test
Patterns: Refactoring Test Code.
Addison-Wesley Signature Series,
2007
Gregory, Janet, Lisa Crispin. Agile
Testing: A Practical Guide for
Testers and Agile Teams.
Addison-Wesley Signature Series,
2008

References
[1] “eXtreme Rules of the Road: How a tester can steer an eXtreme
Programming project toward success”, Lisa Crispin, STQE Jul/Aug
2001
[2] “Testing Extreme Programming”, Lisa Crispin and Tip House,
2003, Addison Wesley
[3] “Continuous integration”,
http://en.wikipedia.org/wiki/Continuous_Integration
[4] “Continuous Integration”, Martin Fowler,
http://www.martinfowler.com/articles/continuousIntegration.html
[5] “Code the Unit Test First”,
http://www.extremeprogramming.org/rules/testfirst.html

Questions???