Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enforcing Quality with DevOps Pipeline Gates


Published on

Introductory presentation regarding DevOps, Continuous Quality/automated testing. Presented at the 2019 CMMI Capability Counts conference (April 30, 2019).

Published in: Business
  • Be the first to comment

Enforcing Quality with DevOps Pipeline Gates

  1. 1. Copyright © 2019 CMMI Institute. All rights reserved. M i c h a e l K i n g C h i e f T e c h n o l o g y O f f i c e r H a l f a k e r a n d A s s o c i a t e s Enforcing Quality with DevOps Pipeline Gates
  2. 2. Agenda  Context  Background: Software Testing Concepts  Background: What is DevOps?  What is Continuous Quality (CQ)?  CQ Concept of Operations  Categories of CQ Tests  CQ Technologies  What Happened?  What did we Learn?  Lessons Learned
  3. 3. Context  Halfaker and Associates (Halfaker, is a midsize company, headquartered in Arlington  Halfaker creates, modernizes, integrates, and secures mission critical systems for Federal Government organizations  Halfaker is a fast-growing organization that invests in improving our Agile management and software engineering maturity to better serve our customers  As Halfaker has scaled rapidly (100 employees in 2015) and begun managing large, more complex, more visible programs, we’ve needed to improve how we ensure quality in our technology solutions  We’ve been prioritizing investment at the intersection of Software Testing/Quality and DevOps
  4. 4. Background: Software Testing Domain The Test Pyramid Diagrams from and  Similar to the Food Pyramid, a healthy Testing Program (“diet”) should be balanced – Mike Cohn and Martin Fowler have written about the concept of the Test Pyramid many times  Some teams focus too much on end-to-end testing and exploratory (ad-hoc) testing, without building the solid foundation of unit and integration tests  Unit tests are small, in-code tests, which developers should write consistently throughout the development and maintenance of software – they are fast to run (because they are often automated) and provide a solid foundation by ensuring that micro-level assumptions are met  Moving up the test pyramid, there should be fewer tests created at each level, with a focus more and more at the macro level of the system  Across the layers, think about: (1) Does it do what it’s supposed to do? (2) Is it intuitive to use correctly? (3) Can I break the system?
  5. 5. Background: What is DevOps?  Development Operations (DevOps) is a set of practices focused on improving the intersection/relationship between software development and IT operations teams  DevOps is based on a culture of collaboration, instead of traditional silos  DevOps is not a just technology – it is more of a philosophy, similar to how Agile is much more than just Scrum ceremonies/tools/roles  DevOps is enabled through the use of technologies that automate activities including building, testing, and delivering software to create smaller, lower risk feedback loops related to software engineering  and The Phoenix Project are great introductions to the topic
  6. 6. What is Continuous Quality (CQ)?  Viewing software testing and quality management as an event done by people with “test” in their job titles doesn’t scale with the velocity and Agility needed for today’s engineering work  Instead, organizations should view Continuous Quality (CQ) as the goal – with a focus on:  Shift Left – Focus on measuring/assessing quality as early as possible, continually “shifting left” to reduce the time a defect/misunderstanding exists  Automate Quality – The more we can automate quality, the more quickly and comprehensively we can assess quality, freeing up time to focus on doing a faster and better job of the quality work that can’t be automated  (If teams don’t have a mature, comprehensive testing program, they certainly need to build that foundation BEFORE they can shift left and automate)
  7. 7. Concept of CQ Gates  We enforce quality in a Continuous Quality environment by defining automated gates that does not let code/software pass without passing the test  These gates enforce rules, such as:  Unit Test Coverage (Goal is 100%, but that isn’t always reasonable, so we set an enforced threshold, such as >= 80% of lines are covered by unit tests; see  Unit Test Pass Rate (e.g. 100% of unit tests must pass)  Static Code Analysis Thresholds (e.g. SonarQube tests re: code complexity)  Accessibility Tests (e.g. Section 508,  Source code vulnerability scan (Look for security weaknesses)  Performance Testing (Build the application and test it for responsiveness)  Capitol One has a blog post about their 16 Gates pipeline-topo-pal-833d15edf0bd
  8. 8. Visualizing Continuous Quality  DevOps Teams should select the gates that provide the best return (improving software quality) on investment of configuration and maintenance
  9. 9. What Happened?  Halfaker established automated Continuous Integration/Continuous Deployment (CI/CD) pipelines across several programs, using the open source Jenkins automation technology  Halfaker then added Continuous Quality gates, which has led to dramatically improved efficiency (less rework and surprises late in the process) and improved quality  We initially focused on a few key gates (Unit Tests, Static Code Analysis (Code Complexity) and Source Code Security Scans), because they were our biggest drivers in ROI  One of our Federal Government customers had an Production Readiness Review (PRR) security audit done by a IV&V team on one of our programs, and they were shocked that we had zero identified security vulnerabilities – something that is very uncommon for this Federal agency  We also standardizing on a source code branching strategy (GitFlow,, which improved the quality of our build process
  10. 10. Some Recommended Technologies  These are some of the technologies we’ve found success with Test Type CQ Approach User Acceptance Testing (UAT) N/A (Humans presenting to customers) Exploratory N/A (Humans exploring) End to End (Functional) Many options such as Cucumber, Gherkin, Protractor Performance Apache JMeter™ Security Fortify or SonarQube static code analysis Code Complexity SonarQube Quality Gate Component (Functional) N/A (Humans exploring parts of the system) Integration Jenkins Build/Delivery Validation Unit SonarQube Quality Gate: Unit Test Coverage >= 80%
  11. 11. Jenkins Automation Server Technology  Jenkins, an open source technology, provides the central orchestration required to define the CI/CD pipeline, which collects, builds, validates quality gates, and deploys the software
  12. 12. SonarQube Continuous Code Quality Tool  SonarQube, an open source technology, provides a platform to analyzing code quality to identify areas for developers to review – below are screenshots showing code quality (left) and quality gates (right)
  13. 13. Example of Unit Test Coverage Analysis Image from code-coverage-unittests-integrationtests.html
  14. 14. Example of Static Code Analysis (Code Complexity) Failure  xxxx Image from
  15. 15. Example of Source Code Security Scan Failure  xxxx Image from
  16. 16. What did we Learn? Make it Easy to See Continuous Quality  Make information convenient (information radiators) – we used open source tools including Capital One’s Hygieia (see right) and Grafana for several of our projects to visualize data related to Continuous Quality and DevOps  Many teams used shared TVs to keep this information very visible for the team, and created some healthy peer pressure not to be the person who broke the build
  17. 17. Example DevOps and Continuous Quality Technologies Planning and Rqmts. (PP, RD, REQM) • Atlassian Jira • Zephyr • CA Agile (Rally) Configuration Management (CM) • Source Repository • GitHub • Bitbucket • Infrastructure Configuration: • Chef • Ansible • Terraform Product Integration (PI) and Release Management • Jenkins CI/CD Pipeline • CA Automic • IBM UrbanCode Testing (VAL, VER) • Unit Testing (JUnit, HtmlUnit, Microsoft Unit Test) • Behavior Driven Dev (Cucumber, Gherkin, RSpec) • Functional Tests (Selenium, Katalon, Appium, Protractor, SmartBear TestComplete) • Accessibility (Deque, aXe, SortSite) • Performance (LoadUI, JMeter, MS VSTS, LoadRunner)
  18. 18. Recap Invest time in improving to get out of reactive firefighting 1. Shift Left! Try to enforce and test as early as you can, as it only gets more expensive the farther requirements, designs, code, etc. make it in the process 2. Automate everything you can and then tune to the right amount of automation – you can’t, and shouldn’t, automate everything; but it’s wise to try and see what sticks! 3. Enforce Continuous Quality gates that require developers to meet or exceed your requirements, which reduces rework and clarifies engineering expectations at the beginning 4. Maximizing the value produced (functional code that comes through the CI/CD pipeline), instead of trying to keep everyone on your teams busy (see The Goal)
  19. 19. Questions? Follow-up Questions? Want to Connect?   @mikehking (Twitter) 
  20. 20. Copyright © 2019 CMMI Institute. All rights reserved. Thank You! Please rate this presentation! C M M I I N S T I T U T E . C O M / C O N F E R E N C E S