Introductory presentation regarding DevOps, Continuous Quality/automated testing. Presented at the 2019 CMMI Capability Counts conference (April 30, 2019).

1. Copyright © 2019 CMMI Institute. All rights reserved.
M i c h a e l K i n g
C h i e f T e c h n o l o g y O f f i c e r
H a l f a k e r a n d A s s o c i a t e s
Enforcing Quality with
DevOps Pipeline Gates

2. Agenda
 Context
 Background: Software Testing Concepts
 Background: What is DevOps?
 What is Continuous Quality (CQ)?
 CQ Concept of Operations
 Categories of CQ Tests
 CQ Technologies
 What Happened?
 What did we Learn?
 Lessons Learned

3. Context
 Halfaker and Associates (Halfaker,
www.halfaker.com) is a midsize company,
headquartered in Arlington
 Halfaker creates, modernizes, integrates, and
secures mission critical systems for Federal
Government organizations
 Halfaker is a fast-growing organization that invests
in improving our Agile management and software
engineering maturity to better serve our customers
 As Halfaker has scaled rapidly (100 employees in
2015) and begun managing large, more complex,
more visible programs, we’ve needed to improve
how we ensure quality in our technology solutions
 We’ve been prioritizing investment at the
intersection of Software Testing/Quality and
DevOps

4. Background: Software Testing Domain
The Test Pyramid
Diagrams from https://martinfowler.com/bliki/TestPyramid.html and
https://martinfowler.com/articles/microservice-testing/#conclusion-test-pyramid
 Similar to the Food Pyramid, a healthy Testing Program (“diet”)
should be balanced – Mike Cohn and Martin Fowler have written
about the concept of the Test Pyramid many times
 Some teams focus too much on end-to-end testing and
exploratory (ad-hoc) testing, without building the solid foundation
of unit and integration tests
 Unit tests are small, in-code tests, which developers should write
consistently throughout the development and maintenance of
software – they are fast to run (because they are often automated)
and provide a solid foundation by ensuring that micro-level
assumptions are met
 Moving up the test pyramid, there should be fewer tests created at
each level, with a focus more and more at the macro level of the
system
 Across the layers, think about: (1) Does it do what it’s supposed to
do? (2) Is it intuitive to use correctly? (3) Can I break the system?

5. Background: What is DevOps?
 Development Operations (DevOps) is a set of practices focused on improving
the intersection/relationship between software development and IT operations
teams
 DevOps is based on a culture of collaboration, instead of traditional silos
 DevOps is not a just technology – it is more of a philosophy, similar to how
Agile is much more than just Scrum ceremonies/tools/roles
 DevOps is enabled through the use of technologies that automate activities
including building, testing, and delivering software to create smaller, lower
risk feedback loops related to software engineering
 Atlassian.com/devops and The Phoenix Project are great introductions to the
topic

6. What is Continuous Quality (CQ)?
 Viewing software testing and quality management as an event done by people
with “test” in their job titles doesn’t scale with the velocity and Agility needed for
today’s engineering work
 Instead, organizations should view Continuous Quality (CQ) as the goal – with a
focus on:
 Shift Left – Focus on measuring/assessing quality as early as possible,
continually “shifting left” to reduce the time a defect/misunderstanding exists
 Automate Quality – The more we can automate quality, the more quickly and
comprehensively we can assess quality, freeing up time to focus on doing a
faster and better job of the quality work that can’t be automated
 (If teams don’t have a mature, comprehensive testing program, they certainly
need to build that foundation BEFORE they can shift left and automate)

7. Concept of CQ Gates
 We enforce quality in a Continuous Quality environment by
defining automated gates that does not let code/software pass
without passing the test
 These gates enforce rules, such as:
 Unit Test Coverage (Goal is 100%, but that isn’t always reasonable, so we
set an enforced threshold, such as >= 80% of lines are covered by unit tests;
see https://martinfowler.com/bliki/TestCoverage.html)
 Unit Test Pass Rate (e.g. 100% of unit tests must pass)
 Static Code Analysis Thresholds (e.g. SonarQube tests re: code complexity)
 Accessibility Tests (e.g. Section 508,
https://www.deque.com/tools/worldspace-attest/)
 Source code vulnerability scan (Look for security weaknesses)
 Performance Testing (Build the application and test it for responsiveness)
 Capitol One has a blog post about their 16 Gates
https://medium.com/capital-one-tech/focusing-on-the-devops-
pipeline-topo-pal-833d15edf0bd

8. Visualizing Continuous Quality
 DevOps Teams should select the gates that provide the best return
(improving software quality) on investment of configuration and
maintenance

9. What Happened?
 Halfaker established automated Continuous Integration/Continuous Deployment
(CI/CD) pipelines across several programs, using the open source Jenkins automation
technology
 Halfaker then added Continuous Quality gates, which has led to dramatically
improved efficiency (less rework and surprises late in the process) and improved
quality
 We initially focused on a few key gates (Unit Tests, Static Code Analysis (Code
Complexity) and Source Code Security Scans), because they were our biggest
drivers in ROI
 One of our Federal Government customers had an Production Readiness Review
(PRR) security audit done by a IV&V team on one of our programs, and they were
shocked that we had zero identified security vulnerabilities – something that is very
uncommon for this Federal agency
 We also standardizing on a source code branching strategy (GitFlow,
https://nvie.com/posts/a-successful-git-branching-model/), which improved the quality
of our build process

10. Some Recommended Technologies
 These are some of the technologies we’ve found success with
Test Type CQ Approach
User Acceptance Testing
(UAT)
N/A (Humans presenting to customers)
Exploratory N/A (Humans exploring)
End to End (Functional) Many options such as Cucumber, Gherkin, Protractor
Performance Apache JMeter™
Security Fortify or SonarQube static code analysis
Code Complexity SonarQube Quality Gate
Component (Functional) N/A (Humans exploring parts of the system)
Integration Jenkins Build/Delivery Validation
Unit SonarQube Quality Gate: Unit Test Coverage >= 80%

11. Jenkins Automation Server Technology
 Jenkins, an open source technology, provides the central orchestration required to
define the CI/CD pipeline, which collects, builds, validates quality gates, and
deploys the software

12. SonarQube Continuous Code Quality Tool
 SonarQube, an open source technology, provides a platform to analyzing code
quality to identify areas for developers to review – below are screenshots showing
code quality (left) and quality gates (right)

13. Example of Unit Test Coverage Analysis
Image from https://blog.lgohlke.de/java/sonarqube/coverage/maven/2015/02/05/sonarqube-
code-coverage-unittests-integrationtests.html

14. Example of Static Code Analysis (Code Complexity) Failure
 xxxx
Image from https://www.sonarqube.org/features/issues-tracking/

15. Example of Source Code Security Scan Failure
 xxxx
Image from https://www.sonarqube.org/features/issues-tracking/

16. What did we Learn?
Make it Easy to See Continuous Quality
 Make information convenient
(information radiators) – we used
open source tools including Capital
One’s Hygieia (see right) and
Grafana for several of our projects
to visualize data related to
Continuous Quality and DevOps
 Many teams used shared TVs to
keep this information very visible
for the team, and created some
healthy peer pressure not to be the
person who broke the build

17. Example DevOps and Continuous Quality Technologies
Planning and Rqmts.
(PP, RD, REQM)
• Atlassian Jira
• Zephyr
• CA Agile (Rally)
Configuration
Management (CM)
• Source Repository
• GitHub
• Bitbucket
• Infrastructure
Configuration:
• Chef
• Ansible
• Terraform
Product Integration
(PI) and Release
Management
• Jenkins CI/CD
Pipeline
• CA Automic
• IBM UrbanCode
Testing (VAL, VER)
• Unit Testing (JUnit,
HtmlUnit, Microsoft
Unit Test)
• Behavior Driven Dev
(Cucumber, Gherkin,
RSpec)
• Functional Tests
(Selenium, Katalon,
Appium, Protractor,
SmartBear
TestComplete)
• Accessibility (Deque,
aXe, SortSite)
• Performance
(LoadUI, JMeter, MS
VSTS, LoadRunner)

18. Recap
Invest time in improving to get out of reactive firefighting
1. Shift Left! Try to enforce and test as early as you can, as it only gets
more expensive the farther requirements, designs, code, etc. make it in
the process
2. Automate everything you can and then tune to the right amount of
automation – you can’t, and shouldn’t, automate everything; but it’s wise
to try and see what sticks!
3. Enforce Continuous Quality gates that require developers to meet or
exceed your requirements, which reduces rework and clarifies
engineering expectations at the beginning
4. Maximizing the value produced (functional code that comes through the
CI/CD pipeline), instead of trying to keep everyone on your teams busy
(see The Goal)

19. Questions?
Follow-up Questions? Want to Connect?
 michael.king@halfaker.com
 @mikehking (Twitter)
 https://www.linkedin.com/in/mikehking

20. Copyright © 2019 CMMI Institute. All rights reserved.
Thank You!
Please rate this presentation!
C M M I I N S T I T U T E . C O M / C O N F E R E N C E S