The document describes Coveros' approach to continuous delivery using a delivery pipeline. It divides the pipeline into three stages - commit stage, acceptance stage, and end game. The commit stage focuses on getting rapid feedback to developers and includes unit testing and static analysis. The acceptance stage includes more extensive quality testing to determine if a change is viable for production. The end game only begins once a release is confident and includes final non-functional testing before deployment. The goal is to balance early rapid feedback with avoiding late surprises by doing just enough testing at each stage.
Thanks for the opportunity to speak today. My name is Gene Gotimer. I’m a senior architect with Coveros, and I’m going to talk about developing your delivery pipeline. Specifically, I’m going to talk about where to put different types of testing into your process to make sure your pipeline is efficient and effective. Hopefully, this will be useful whether you are doing continuous delivery or not, whether you have a lot of automation or are doing things largely manually.
The continuous delivery pipeline is the process of taking new or changed features from developers, and getting features deployed into production and delivered quickly to the customer. Gene Gotimer says testing within continuous delivery pipelines should be designed so the earliest tests are the quickest and easiest to run, giving developers the fastest feedback. Successive rounds of testing lead to increased confidence that the code is a viable candidate for production and that more expensive tests—time, effort, cost—are justified. Manual testing is performed toward the end of the pipeline, leaving computers to do as much work as possible before people get involved. Although it is tempting to arrange the delivery pipeline in phases (e.g., functional tests, then acceptance tests, then load and performance tests, then security tests), this can lead to serious problems progressing far down the pipeline before they are caught. Gene shows how to arrange your tests so each round provides just enough testing to give you confidence that the next set of tests is worth the investment. He explores how to get the right types of testing into your pipeline at the right points.
Coveros is a consulting company that helps organizations build better software. We provide software development, application security, QA/testing, and software process improvement services. Coveros focuses on organizations that must build and deploy software within the constraints of significant regulatory or compliance requirements. The primary markets we serve include: DoD, Homeland Security & associated critical infrastructure companies, Healthcare providers, and Financial services institutions
These are some of the organization’s our team and corporate executives have helped build better software. While many of these organization’s are large, we have significant experience helping orgs of all size address software challenges.
Doesn’t have to be automated. More automated is better, but we have a delivery pipeline anyway.
The goal of the delivery pipeline is to build confidence that we have a viable candidate for production.
The further you get through the pipeline, the more expensive the stage gates are to pass: the tests are harder to set up and take longer to run. That means feedback take longer to get.
Conversely, the closer to the front of the pipeline, the tests are quicker and easier and will be run far more often. And the feedback will be available that much quicker.
Invest first in automation where it is easier and will run more often. You’ll get the most obvious payoff.
Thoughtworks suggests adding all of these steps to you CI engine, even if they are manual. Then you are reminded that a manual step has to take place to move to the next step.
Developer centric
The automated build is critical. It has to happen so often that there is no doubt that it must be automated, no matter how easy it is to do manually. No questions asked– automate the build first.
Remember, we want to get a quick level of confidence that these changes represent a viable production candidate, and that the time and effort of running further tests and checks is warranted.
Code is checked in.
That triggers an automated
build, unit tests, static analysis,
and packaging for deploy.
If everything passes, deploy to test.
If not, back to coding.
A deploy to test triggers a smoke tests, integration tests, one or more rounds of functional tests, regression tests, possibly more deploys and smoke tests, and finally acceptance tests.
The developers have not shifted modes – work on the commit stage is still going on. Since we got through the early quality gates, we are confident that running these next sets of tests is worth while even if they take more time. But we can’t stop everything just to watch if the code gets through this next round of tests.
The team is generally not waiting for this stage to pass before continuing work on other features, but will still make it a priority to resolve any problems that are found during this stage.
We are confident we have a viable production candidate.
This includes “packaging”, maybe marketing, documentation, other non-development-type stuff.
These tests might be more expensive: time, effort, manual inspection, monopolizing an environment for an extended time, could be outsourcing to cloud (e.g., LoadStorm, Sauce Labs) or bringing in specialists (e.g., security) so it could be actual money.
But no surprises, so we should already expect that these tests are going to succeed.
List of funny comments in source code:
“When I wrote this, only God and I understood what I was doing. Now, God only knows.”
― Karl Weierstrass, German mathematician that lived in the late 1800s
No sense doing any other type of testing on the deployed system if we don’t know if the deployment was successful.
How many times have you found all sorts of bugs, wondered how this code ever got out of development because it just doesn’t work, only to find out that a step in the deployment was left out or a configuration setting is wrong. You wasted all that time testing a defective product.
Do just enough testing to be sure that further testing is justified.
Often subjective for interpretation and/or evaluation. That means manual and that means it will take time.
Testing for quality characteristics that do not map to functional requirements. Other non-functional requirements need to be tested on an entire system operating on a production-like environment
http://www.stellman-greene.com/blog/wp-content/uploads/2009/10/jeez.-lady.png