Downloaded 22 times
Uploaded byGene Gotimer
Testing in a Continuous Delivery Pipeline - Better, Faster, Cheaper
The document outlines Coveros' approach to building security-critical applications using agile methods, detailing their various services related to agile development, security analysis, and training. It explains the delivery pipeline process from code change to production, highlighting the importance of early testing and feedback to prevent surprises. Different stages of testing including commit, acceptance, and end game are discussed, emphasizing the balance between rapid feedback, quality assurance, and final delivery preparations.
More Related Content
Similar to Testing in a Continuous Delivery Pipeline - Better, Faster, Cheaper
Testing in a Continuous Delivery Pipeline - Better, Faster, Cheaper
- 1. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 1@CoverosGene Agility. Security. Delivered. Gene Gotimer Senior Architect
- 2. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 2@CoverosGene About Coveros • Coveros builds security-critical applications using agile methods. • Coveros Services • Agile transformations • Agile development and testing • DevOps and continuous integration • Application security analysis • Agile & Security training • Government qualifications • DCAA approved rates and accounting • TS facility clearance Areas of Expertise
- 3. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 3@CoverosGene Select Clients
- 4. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 4@CoverosGene Delivery Pipeline Process of taking a code change from developers and getting it deployed into production or delivered to the customer • Stages along the way • Later stages lead • to higher confidence • closer to production
- 5. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 5@CoverosGene Delivery Pipeline Do we have a viable candidate for production?
- 6. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 6@CoverosGene Delivery Pipeline Requirement Code Check-in Unit Tests Deploy to Test Functional Tests Deploy to Staging Acceptance Tests Deploy to Pre-Prod Quality Gate Trigger Performance Tests Security Tests Deploy to Prod Rapid Feedback No surprises
- 7. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 7@CoverosGene Goal is to Balance Early Rapid Feedback No Late Surprises
- 8. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 8@CoverosGene Everything Can’t Be First Do just enough of each type of testing early in the pipeline to determine if further testing is justified.
- 9. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 9@CoverosGene
- 10. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 10@CoverosGene Value Stream • List out steps from developer to production • That is the delivery pipeline • whether manual or automated • Identify time for each step • execution time • wait time • Helps show • where bottlenecks are • what should be automated
- 11. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 11@CoverosGene Pipeline Stages • Not hard-and-fast stages • Gradual change in focus
- 12. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 12@CoverosGene Commit Stage Commit Stage Requirement Code Check-in Unit Tests Deploy to Test Functional Tests Deploy to Staging Acceptance Tests Deploy to Pre-Prod Performance Tests Security Tests Deploy to Prod • Code-focused • Rapid feedback • 10 minutes maximum • Developers are waiting
- 13. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 13@CoverosGene Acceptance Stage Acceptance Stage Requirement Code Check-in Unit Tests Deploy to Test Functional Tests Deploy to Staging Acceptance Tests Deploy to Pre-Prod Performance Tests Security Tests Deploy to Prod • Quality-focused • Is this is a viable candidate for production?
- 14. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 14@CoverosGene End Game End Game Requirement Code Check-in Unit Tests Deploy to Test Functional Tests Deploy to Staging Acceptance Tests Deploy to Pre-Prod Performance Tests Security Tests Deploy to Prod • Delivery-focused • Steps that only get done when we are releasing • Does not begin until you are confident there will be no surprises
- 15. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 15@CoverosGene Pipeline Steps Commit Stage • Compile • Unit tests • Static analysis Acceptance Stage • Functional tests • Regression tests • Acceptance tests • System integration • Security testing • Performance testing • Exploratory testing • Usability testing End Game • Security testing • Performance testing • Exploratory testing • Usability testing • Packaging • Printed documentation • Release announcement
- 16. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 16@CoverosGene Pipeline Steps Commit Stage • Compile • Unit tests • Static analysis Acceptance Stage • Functional tests • Regression tests • Acceptance tests • System integration • Some security testing • Performance trend • Early exploratory testing • Basic usability testing End Game • Mandated security test • Full load and performance test • Continuing exploratory testing • Focus group usability testing • Packaging • Printed documentation • Release announcement Do just enough testing to determine if further testing is justified.
- 17. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 17@CoverosGene Example: Performance Testing • Short JMeter test • On development system, no isolation • 10 concurrent users for 10,000 requests • Track the trend • Answers: “Are we getting slower or faster?” • Full load and performance test • Dedicated environment, no other traffic • Production-sized servers • 1,000 concurrent users for 4 hours • Answers: “What is the sustained capacity and throughput?”
- 18. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 18@CoverosGene Example: Security Testing • Functional tests run through OWASP ZAP proxy • During early testing • Piggy-back on existing testing • Answers: “Do we have any XSS vulnerabilities?” • OpenVAS system scanning • Weekly in test environment • Looks for open network ports • Looks for software with CVEs • Answers: “Is Nessus likely to find anything?” • HP WebInspect application security scanning • By corporate security group • Looks for black-box web vulnerabilities • Answers: “Do we have any XSS vulnerabilities?” • Nessus system scanning • By corporate security group • Looks for open network ports • Looks for software with CVEs • Answers: “Is system compliant?”
- 19. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 19@CoverosGene Advantages of Earlier Testing • Quicker feedback cycle • Easier to fix problems that are found • Developer still has context of changes • Less rework on defective product • Proactive response, not reactive
- 20. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 20@CoverosGene Code-focused
- 21. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 21@CoverosGene Testing in the Commit Stage • Code-focused • Developer-centric • Rapid feedback • Developer waits until complete • 10 minutes maximum Types of Activities • Continuous integration • Compile • Unit tests • Static analysis • Dependency analysis
- 22. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 22@CoverosGene Unit Testing • Unit testing is not QA! • Developer tool • Early confirmation of code behavior • Executable documentation • Fearless refactoring
- 23. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 23@CoverosGene Code Coverage • A tool, not a target • Measures code executed while unit tests running • NOT amount of code tested • Not covered = not tested • Covered = possibly tested
- 24. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 24@CoverosGene Mutation Testing • Reruns unit tests against modified versions of your code • If tests still pass, code isn’t tested • Tests quality of tests public int foo(int i) { i--; return i; } public int foo(int i) { i++; return i; }
- 25. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 25@CoverosGene Static Analysis • Early detection of coding issues • style issues • duplicate code blocks • declared but unused variables • confusing code • race conditions • SQL injection • resource leaks
- 26. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 26@CoverosGene Third-party Components OWASP Top 10 2013: A9-Using Components with Known Vulnerabilities Scan your third-party libraries Update proactively, not reactively
- 27. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 27@CoverosGene Quality-focused
- 28. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 28@CoverosGene Testing in the Acceptance Stage • Quality-focused • Bulk of the pipeline • Until confident that you have a viable candidate for production Types of Activities • Functional tests • Regression tests • Acceptance tests • System integration • Some security testing • Performance trend • Early exploratory testing • Basic usability testing
- 29. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 29@CoverosGene Automated Deployment • Repeatable, reliable deployments • Test that through practice • Same deploy process everywhere • You will find more reasons to deploy
- 30. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 30@CoverosGene Smoke Testing • After every deployment • Must be quick • Test the deployment, not the functionality • Focus on • basic signs of life • interfaces between systems • configuration settings
- 31. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 31@CoverosGene Delivery-focused
- 32. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 32@CoverosGene Testing in the End Game • Delivery-focused • Steps that only get done when we are releasing • Tests that are too expensive to do every build • time • resources • effort • Don’t start the End Game until you are confident you won’t be surprised Types of Activities • Non-functional tests • Mandated security testing • Full load and performance test • Continuing exploratory testing • Focus group usability testing • Packaging • Printed documentation • Release announcement
- 33. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 33@CoverosGene Non-functional Testing • Availability testing • Accessibility testing • Baseline testing • Compatibility testing • Compliance testing • Configuration testing • Documentation testing • Endurance testing • Ergonomics testing • Interoperability testing • Installation testing • Internationalization testing • Load testing • Localization testing • Maintainability testing • Operational readiness testing • Performance testing • Portability testing • Recovery testing • Reliability testing • Resilience testing • Scalability testing • Security testing • Stability testing • Stress testing • Supportability testing • Testability testing • Usability testing • Volume testing Image by Andrew Stellman via http://www.stellman-greene.com/2010/02/17/nonfunctional-requirements-qa/
- 34. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 34@CoverosGene Parallel Testing • Conduct long-running tests in parallel • Upside: less elapsed time • Downside: no feedback between tests • Should already be an expectation that these tests will pass
- 35. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 35@CoverosGene
- 36. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 36@CoverosGene Summary • Early rapid feedback vs. no late surprises • Do just enough of each type of testing early in the pipeline to determine if further testing is justified
- 37. © COPYRIGHT 2016COVEROS, INC. ALL RIGHTS RESERVED. 37@CoverosGene Questions? Gene Gotimer gene.gotimer@coveros.com @CoverosGene
Editor's Notes
- #2 Thanks for the opportunity to speak today. My name is Gene Gotimer. I’m a senior architect with Coveros, and I’m going to talk about developing your delivery pipeline. Specifically, I’m going to talk about where to put different types of testing into your process to make sure your pipeline is efficient and effective. Hopefully, this will be useful whether you are doing continuous delivery or not, whether you have a lot of automation or are doing things largely manually. The continuous delivery pipeline is the process of taking new or changed features from developers, and getting features deployed into production and delivered quickly to the customer. Gene Gotimer says testing within continuous delivery pipelines should be designed so the earliest tests are the quickest and easiest to run, giving developers the fastest feedback. Successive rounds of testing lead to increased confidence that the code is a viable candidate for production and that more expensive tests—time, effort, cost—are justified. Manual testing is performed toward the end of the pipeline, leaving computers to do as much work as possible before people get involved. Although it is tempting to arrange the delivery pipeline in phases (e.g., functional tests, then acceptance tests, then load and performance tests, then security tests), this can lead to serious problems progressing far down the pipeline before they are caught. Gene shows how to arrange your tests so each round provides just enough testing to give you confidence that the next set of tests is worth the investment. He explores how to get the right types of testing into your pipeline at the right points.
- #3 Coveros is a consulting company that helps organizations build better software. We provide software development, application security, QA/testing, and software process improvement services. Coveros focuses on organizations that must build and deploy software within the constraints of significant regulatory or compliance requirements. The primary markets we serve include: DoD, Homeland Security & associated critical infrastructure companies, Healthcare providers, and Financial services institutions
- #4 These are some of the organization’s our team and corporate executives have helped build better software. While many of these organization’s are large, we have significant experience helping orgs of all size address software challenges.
- #5 Doesn’t have to be automated. More automated is better, but we have a delivery pipeline anyway.
- #6 The goal of the delivery pipeline is to build confidence that we have a viable candidate for production.
- #7 The further you get through the pipeline, the more expensive the stage gates are to pass: the tests are harder to set up and take longer to run. That means feedback take longer to get. Conversely, the closer to the front of the pipeline, the tests are quicker and easier and will be run far more often. And the feedback will be available that much quicker. Invest first in automation where it is easier and will run more often. You’ll get the most obvious payoff.
- #11 Thoughtworks suggests adding all of these steps to you CI engine, even if they are manual. Then you are reminded that a manual step has to take place to move to the next step.
- #13 Developer centric The automated build is critical. It has to happen so often that there is no doubt that it must be automated, no matter how easy it is to do manually. No questions asked– automate the build first. Remember, we want to get a quick level of confidence that these changes represent a viable production candidate, and that the time and effort of running further tests and checks is warranted. Code is checked in. That triggers an automated build, unit tests, static analysis, and packaging for deploy. If everything passes, deploy to test. If not, back to coding.
- #14 A deploy to test triggers a smoke tests, integration tests, one or more rounds of functional tests, regression tests, possibly more deploys and smoke tests, and finally acceptance tests. The developers have not shifted modes – work on the commit stage is still going on. Since we got through the early quality gates, we are confident that running these next sets of tests is worth while even if they take more time. But we can’t stop everything just to watch if the code gets through this next round of tests. The team is generally not waiting for this stage to pass before continuing work on other features, but will still make it a priority to resolve any problems that are found during this stage.
- #15 We are confident we have a viable production candidate. This includes “packaging”, maybe marketing, documentation, other non-development-type stuff. These tests might be more expensive: time, effort, manual inspection, monopolizing an environment for an extended time, could be outsourcing to cloud (e.g., LoadStorm, Sauce Labs) or bringing in specialists (e.g., security) so it could be actual money. But no surprises, so we should already expect that these tests are going to succeed.
- #26 List of funny comments in source code: “When I wrote this, only God and I understood what I was doing. Now, God only knows.” ― Karl Weierstrass, German mathematician that lived in the late 1800s
- #31 No sense doing any other type of testing on the deployed system if we don’t know if the deployment was successful. How many times have you found all sorts of bugs, wondered how this code ever got out of development because it just doesn’t work, only to find out that a step in the deployment was left out or a configuration setting is wrong. You wasted all that time testing a defective product. Do just enough testing to be sure that further testing is justified.
- #34 Often subjective for interpretation and/or evaluation. That means manual and that means it will take time. Testing for quality characteristics that do not map to functional requirements. Other non-functional requirements need to be tested on an entire system operating on a production-like environment http://www.stellman-greene.com/blog/wp-content/uploads/2009/10/jeez.-lady.png