Enabling Agile Testing Through Continuous Integration Agile2009
Upcoming SlideShare
Loading in...5
×
 

Enabling Agile Testing Through Continuous Integration Agile2009

on

  • 4,487 views

A Continuous Integration system is often considered one of the key elements involved in supporting an agile software development and testing environment. As a traditional software tester transitioning ...

A Continuous Integration system is often considered one of the key elements involved in supporting an agile software development and testing environment. As a traditional software tester transitioning to an agile development environment it became clear to me that I would need to put this essential infrastructure in place and promote improved development practices in order to make the transition to agile testing possible. This experience report discusses a continuous integration implementation I led last year. The initial motivations for implementing continuous integration are discussed and a pre and post-assessment using Martin Fowler's "Practices of Continuous Integration" is provided along with the technical specifics of the implementation. The report concludes with a retrospective of my experiences implementing and promoting continuous integration within the context of agile testing.

Statistics

Views

Total Views
4,487
Views on SlideShare
4,466
Embed Views
21

Actions

Likes
7
Downloads
255
Comments
0

2 Embeds 21

http://www.slideshare.net 19
http://inside-mleo.blogspot.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Enabling Agile Testing Through Continuous Integration Agile2009 Enabling Agile Testing Through Continuous Integration Agile2009 Presentation Transcript

  • Enabling Agile Testing Through Continuous Integration Sean Stolberg, Pacific Northwest National Laboratory
  • Where We’re Going Today Introduction - Being thrown into the deep end of Agile Testing Our Plan – Deciding to Implement Agile Practices The Problem The Solution - Continuous Integration (CI) What is CI? Implementing CI Assessing the Team’s CI Practices Assessing Our New Agile Development Practices and Testing Capabilities Retrospective
  • Being Thrown Into the Deep End Me  Eight years as traditional software tester/QA engineer.  November 2007 – First job as a tester on an Agile team. First two sprints  Humbling  All the “moves” I knew didn’t work too well.  Kept coming up short on time, test coverage, and accumulation of technical debt. Re-evaluation  “What I’m doing is not working, so what will?”
  • Practices We Decided to Implement 1. Define and execute “just-enough” acceptance tests. [1] 2. Automate as close to 100% of the acceptance tests as possible. [2] 3. Automate acceptance tests using a “subcutaneous” test approach with a xUnit test framework. [2] [1] “eXtreme Rules of the Road: How a tester can steer an eXtreme Programming project toward success”, Lisa Crispin, STQE Jul/Aug 2001 [2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley
  • Practices We Decided to Implement (continued) 5. Run all acceptance tests in the regression test suite with the build, daily (at a minimum). [4] 6. Develop unit tests for all new code during a sprint. [5] 7. Run all unit tests with every build. [4] 8. Run multiple builds per day. [4] [4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html [5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html
  • The Problem We didn’t have an automation framework. How could we:  Implement our new development practices?  Run all acceptance tests in the regression test suite with the build?  Run all unit tests with every build?  Run multiple builds per day?
  • The Solution Continuous Integration  Commonly used by teams already successfully implementing Agile Testing.  Provides framework to implement several of the practices.  We needed to find out more.
  • What is Continuous Integration? Describes a set of software engineering practices that speed up the delivery of software by decreasing integration times Emerged in the XP community. Martin Fowler and Kent Beck – helped make it popular.
  • What CI Might Look Like 1. Automated source repository change detection  Change is detected resulting to a chain of events being put into motion 2. Get latest source 3. Compile (or run through other check; interpreted code) 4. Run unit tests 5. Deploy to test environment 6. Run acceptance tests 7. Send results of testing 8. Publish build if tests pass
  • Speeding Up the Feedback Loop Compile Report Unit Testing Continuous Integration Notify Deploy Acceptance Publish build Testing
  • Tools Used
  • Virtual Server Host • First step in build automation • Configuration • Dual Quad Core Xeon 1.6ghz • 8GB Ram • 4 x 146GB 15k RPM SCSI drives in RAID-5. • Unified build environment and administration • Get build machines out from under the desks and into the server room
  • Virtual Build Server IF a source code change is detect • Refresh source code • Build (ABS using VS2008) • Clean build • Build All • Build Installer IF no build errors were found • Prep the test VM for product install and testing (VBScript helper script called via STAF) • Copy install files to test VM (STAF) • Install app-under-test on test VM (STAF) • Discover and deploy all Unit and Acceptance tests to test VM (custom C# helper app/STAF) • Execute tests on test VM (custom C# helper app/STAF) • Copy test result .xml file(s) from test VM to build machine (STAF) • Process the test result .xml files (custom C# helper app)
  • Communication Send mail with • test results (PASS or FAIL with details of failures) • Link to location of build • Build logs • Change logs
  • Assessing our CI Practices Maintain a Single Source Repository Before CI: Seapine Surround SCM source repository. After CI: No change. Automate the Build Before CI: Some partial automation; still very manual. Heterogeneous build environments. After CI: Yes, with Automated Build Studio (ABS). Make Your Build Self-Testing Before CI: Not self testing. After CI: NUnit framework unit and acceptance tests.
  • Assessing our CI Practices (continued) Everyone Commits Every Day Before CI: Unknown, varied probably. After CI: We can only hope. Every Commit Should Build the Mainline on an Integration Machine Before CI: No, it was not happening. After CI: Yes, ABS's Continuous Integration Tasks helped us do this. Keep the Build Fast Before CI: Multi-pass builds, unordered dependencies. After CI: 10 to 15 min; refactoring needed sooner than later.
  • Assessing our CI Practices (continued) Test in a Clone of the Production Environment Before CI: Yes, but not automated After CI: Using clean virtual machine test clients to install and test Make it Easy for Anyone to Get the Latest Executable Before CI: Not all projects using the common build repository. Some private file share locations for production code After CI: All products building to common location now. Build mail contains link to new build location
  • Assessing our CI Practices (continued) Everyone can see what's happening Before CI: Limited to ad hoc e-mails, no web, no reporting, different project worked differently. After CI: Use ABS's web interface to see the progress of builds, and email build and test status. Automate Deployment Before CI: Not being done. After CI: Yes, automatically deploy the build, then test it.
  • Assessing Our Agile Practices Define and execute “just-enough” acceptance tests  Made acceptance test definition a required task during sprint planning.  Developers and customer liked visibility into testing. Automate as close to 100% of the acceptance tests as possible  Now tried to automate as close to 100% acceptance tests as possible.  Slowed our accumulation of technical test debt.
  • Assessing Our Agile Practices (continued) Automate acceptance tests using a “subcutaneous” test approach with an xUnit test framework  NUnit test framework used to automate our acceptance/functional testing using the API of the application.  Acceptance tests versioned code in the repository and possible to run in a daily build. Run all acceptance tests in the regression test suite with the build, daily (at a minimum)  NUnit acceptance tests now ran with a daily build.  Practice enabled using our new continuous integration system.
  • Assessing Our Agile Practices (continued) Develop unit tests for all new code during a sprint  Developers started putting more emphasis on developing unit tests during the same sprint.  Practice encouraged due to ease with which unit tests could now be run with the build. Run all unit tests with every build  NUnit tests now ran with every build. Run multiple builds per day  Builds started automatically when changes to the source repository were checked in.
  • Retrospective Personal “biggest discoveries”  Testing in sync with development. Embracing fully automated acceptance tests  Exciting and intimidating. Hard to convince developers to go through the automation “hump-of-pain”  But it’s worth it.  Devoting personal time toward investigation and proof-of-concept paid off. Can’t imagine working on a team not using Agile Testing or development  Get “hooked” on injecting testing and quality early into the cycle.  Adds immediate value.
  • Recommended Reading Crispin, Lisa, Tip House. Testing Extreme Programming. Addison- Wesley Professional, 2002 Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code. Addison-Wesley Signature Series, 2007 Gregory, Janet, Lisa Crispin. Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley Signature Series, 2008
  • References [1] “eXtreme Rules of the Road: How a tester can steer an eXtreme Programming project toward success”, Lisa Crispin, STQE Jul/Aug 2001 [2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley [3] “Continuous integration”, http://en.wikipedia.org/wiki/Continuous_Integration [4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html [5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html
  • Questions???