Enabling Agile Testing Through
Continuous Integration
 Sean Stolberg, Pacific Northwest National Laboratory
Where We’re Going Today

 Introduction - Being thrown into the deep end of Agile Testing
 Our Plan – Deciding to Implement...
Being Thrown Into the Deep End


 Me
    Eight years as traditional software tester/QA engineer.
    November 2007 – Fir...
Practices We Decided to Implement


1. Define and execute “just-enough” acceptance tests. [1]
2. Automate as close to 100%...
Practices We Decided to Implement                                                               (continued)




5. Run all...
The Problem


  We didn’t have an automation framework.
  How could we:
     Implement our new development practices?
  ...
The Solution


  Continuous Integration
     Commonly used by teams already successfully implementing
      Agile Testing...
What is Continuous Integration?


  Describes a set of software engineering practices that

             speed up the deli...
What CI Might Look Like

1. Automated source repository change detection
        Change is detected resulting to a chain ...
Speeding Up the Feedback Loop


                                                Compile




                     Report   ...
Tools Used
Virtual Server Host

• First step in build
  automation
• Configuration
  • Dual Quad Core Xeon
      1.6ghz
  • 8GB Ram
 ...
Virtual Build Server

IF a source code change is detect
• Refresh source code
• Build (ABS using VS2008)
        • Clean b...
Communication
Send mail with
• test results (PASS or FAIL
   with details of failures)
• Link to location of build
• Build...
Assessing our CI Practices
 Maintain a Single Source Repository
    Before CI: Seapine Surround SCM source repository.
   ...
Assessing our CI Practices                (continued)


 Everyone Commits Every Day
    Before CI: Unknown, varied probabl...
Assessing our CI Practices                  (continued)




 Test in a Clone of the Production Environment
    Before CI: ...
Assessing our CI Practices                  (continued)




 Everyone can see what's happening
    Before CI: Limited to a...
Assessing Our Agile Practices

  Define and execute “just-enough” acceptance tests
   Made acceptance test definition a r...
Assessing Our Agile Practices                     (continued)




  Automate acceptance tests using a “subcutaneous” test ...
Assessing Our Agile Practices                (continued)




  Develop unit tests for all new code during a sprint
    De...
Retrospective

 Personal “biggest discoveries”
  Testing in sync with development.


 Embracing fully automated acceptanc...
Recommended
Reading
Crispin, Lisa, Tip House. Testing
Extreme Programming. Addison-
Wesley Professional, 2002




Gerard M...
References

 [1] “eXtreme Rules of the Road: How a tester can steer an eXtreme
 Programming project toward success”, Lisa ...
Questions???
Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009
Upcoming SlideShare
Loading in …5
×

Enabling Agile Testing Through Continuous Integration Agile2009

3,795 views

Published on

A Continuous Integration system is often considered one of the key elements involved in supporting an agile software development and testing environment. As a traditional software tester transitioning to an agile development environment it became clear to me that I would need to put this essential infrastructure in place and promote improved development practices in order to make the transition to agile testing possible. This experience report discusses a continuous integration implementation I led last year. The initial motivations for implementing continuous integration are discussed and a pre and post-assessment using Martin Fowler's "Practices of Continuous Integration" is provided along with the technical specifics of the implementation. The report concludes with a retrospective of my experiences implementing and promoting continuous integration within the context of agile testing.

Published in: Technology, Business

Enabling Agile Testing Through Continuous Integration Agile2009

  1. Enabling Agile Testing Through Continuous Integration Sean Stolberg, Pacific Northwest National Laboratory
  2. Where We’re Going Today Introduction - Being thrown into the deep end of Agile Testing Our Plan – Deciding to Implement Agile Practices The Problem The Solution - Continuous Integration (CI) What is CI? Implementing CI Assessing the Team’s CI Practices Assessing Our New Agile Development Practices and Testing Capabilities Retrospective
  3. Being Thrown Into the Deep End Me  Eight years as traditional software tester/QA engineer.  November 2007 – First job as a tester on an Agile team. First two sprints  Humbling  All the “moves” I knew didn’t work too well.  Kept coming up short on time, test coverage, and accumulation of technical debt. Re-evaluation  “What I’m doing is not working, so what will?”
  4. Practices We Decided to Implement 1. Define and execute “just-enough” acceptance tests. [1] 2. Automate as close to 100% of the acceptance tests as possible. [2] 3. Automate acceptance tests using a “subcutaneous” test approach with a xUnit test framework. [2] [1] “eXtreme Rules of the Road: How a tester can steer an eXtreme Programming project toward success”, Lisa Crispin, STQE Jul/Aug 2001 [2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley
  5. Practices We Decided to Implement (continued) 5. Run all acceptance tests in the regression test suite with the build, daily (at a minimum). [4] 6. Develop unit tests for all new code during a sprint. [5] 7. Run all unit tests with every build. [4] 8. Run multiple builds per day. [4] [4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html [5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html
  6. The Problem We didn’t have an automation framework. How could we:  Implement our new development practices?  Run all acceptance tests in the regression test suite with the build?  Run all unit tests with every build?  Run multiple builds per day?
  7. The Solution Continuous Integration  Commonly used by teams already successfully implementing Agile Testing.  Provides framework to implement several of the practices.  We needed to find out more.
  8. What is Continuous Integration? Describes a set of software engineering practices that speed up the delivery of software by decreasing integration times Emerged in the XP community. Martin Fowler and Kent Beck – helped make it popular.
  9. What CI Might Look Like 1. Automated source repository change detection  Change is detected resulting to a chain of events being put into motion 2. Get latest source 3. Compile (or run through other check; interpreted code) 4. Run unit tests 5. Deploy to test environment 6. Run acceptance tests 7. Send results of testing 8. Publish build if tests pass
  10. Speeding Up the Feedback Loop Compile Report Unit Testing Continuous Integration Notify Deploy Acceptance Publish build Testing
  11. Tools Used
  12. Virtual Server Host • First step in build automation • Configuration • Dual Quad Core Xeon 1.6ghz • 8GB Ram • 4 x 146GB 15k RPM SCSI drives in RAID-5. • Unified build environment and administration • Get build machines out from under the desks and into the server room
  13. Virtual Build Server IF a source code change is detect • Refresh source code • Build (ABS using VS2008) • Clean build • Build All • Build Installer IF no build errors were found • Prep the test VM for product install and testing (VBScript helper script called via STAF) • Copy install files to test VM (STAF) • Install app-under-test on test VM (STAF) • Discover and deploy all Unit and Acceptance tests to test VM (custom C# helper app/STAF) • Execute tests on test VM (custom C# helper app/STAF) • Copy test result .xml file(s) from test VM to build machine (STAF) • Process the test result .xml files (custom C# helper app)
  14. Communication Send mail with • test results (PASS or FAIL with details of failures) • Link to location of build • Build logs • Change logs
  15. Assessing our CI Practices Maintain a Single Source Repository Before CI: Seapine Surround SCM source repository. After CI: No change. Automate the Build Before CI: Some partial automation; still very manual. Heterogeneous build environments. After CI: Yes, with Automated Build Studio (ABS). Make Your Build Self-Testing Before CI: Not self testing. After CI: NUnit framework unit and acceptance tests.
  16. Assessing our CI Practices (continued) Everyone Commits Every Day Before CI: Unknown, varied probably. After CI: We can only hope. Every Commit Should Build the Mainline on an Integration Machine Before CI: No, it was not happening. After CI: Yes, ABS's Continuous Integration Tasks helped us do this. Keep the Build Fast Before CI: Multi-pass builds, unordered dependencies. After CI: 10 to 15 min; refactoring needed sooner than later.
  17. Assessing our CI Practices (continued) Test in a Clone of the Production Environment Before CI: Yes, but not automated After CI: Using clean virtual machine test clients to install and test Make it Easy for Anyone to Get the Latest Executable Before CI: Not all projects using the common build repository. Some private file share locations for production code After CI: All products building to common location now. Build mail contains link to new build location
  18. Assessing our CI Practices (continued) Everyone can see what's happening Before CI: Limited to ad hoc e-mails, no web, no reporting, different project worked differently. After CI: Use ABS's web interface to see the progress of builds, and email build and test status. Automate Deployment Before CI: Not being done. After CI: Yes, automatically deploy the build, then test it.
  19. Assessing Our Agile Practices Define and execute “just-enough” acceptance tests  Made acceptance test definition a required task during sprint planning.  Developers and customer liked visibility into testing. Automate as close to 100% of the acceptance tests as possible  Now tried to automate as close to 100% acceptance tests as possible.  Slowed our accumulation of technical test debt.
  20. Assessing Our Agile Practices (continued) Automate acceptance tests using a “subcutaneous” test approach with an xUnit test framework  NUnit test framework used to automate our acceptance/functional testing using the API of the application.  Acceptance tests versioned code in the repository and possible to run in a daily build. Run all acceptance tests in the regression test suite with the build, daily (at a minimum)  NUnit acceptance tests now ran with a daily build.  Practice enabled using our new continuous integration system.
  21. Assessing Our Agile Practices (continued) Develop unit tests for all new code during a sprint  Developers started putting more emphasis on developing unit tests during the same sprint.  Practice encouraged due to ease with which unit tests could now be run with the build. Run all unit tests with every build  NUnit tests now ran with every build. Run multiple builds per day  Builds started automatically when changes to the source repository were checked in.
  22. Retrospective Personal “biggest discoveries”  Testing in sync with development. Embracing fully automated acceptance tests  Exciting and intimidating. Hard to convince developers to go through the automation “hump-of-pain”  But it’s worth it.  Devoting personal time toward investigation and proof-of-concept paid off. Can’t imagine working on a team not using Agile Testing or development  Get “hooked” on injecting testing and quality early into the cycle.  Adds immediate value.
  23. Recommended Reading Crispin, Lisa, Tip House. Testing Extreme Programming. Addison- Wesley Professional, 2002 Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code. Addison-Wesley Signature Series, 2007 Gregory, Janet, Lisa Crispin. Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley Signature Series, 2008
  24. References [1] “eXtreme Rules of the Road: How a tester can steer an eXtreme Programming project toward success”, Lisa Crispin, STQE Jul/Aug 2001 [2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley [3] “Continuous integration”, http://en.wikipedia.org/wiki/Continuous_Integration [4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html [5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html
  25. Questions???

×