Each month more than 124+ million unique visitors access content from USA TODAY and Gannett’s local media organizations, making them the largest US newspaper publisher by total daily circulation. The company’s continuous integration pipeline has evolved from a slow-moving tortoise to a sprinting hare and continues to evolve today. When they started their pipeline, everything was a manual process. Now they have a dedicated operations team that oversees onboarding, maintains the infrastructure, cares for the continuous integration and continuous delivery tools, provides continuous support, and more. The organization has reduced the operational burden and allowed developers to focus on engineering, and the quality engineering team manages the CI platform infrastructure and helps teams optimize the pipeline to build applications, execute quality checks, and deploy with ease. Greg Sypolt will describe how Gannett started from basics and is now able to build, deploy, and maintain their applications through a dedicated platform using Helm, Docker, and Kubernetes. Pipeline ownership should be a community activity, not an individual one. Let's explore the evolution of excellence!

1. The Evolution of a Continuous Integration Pipeline
Greg Sypolt unsplash.com

2. The Scale Of USA TODAY NETWORK
Gannett is a leading local-to-
national media & marketing
solutions company, and the
largest local media company in
America, with 100+ newsrooms
spanning 34 states.
Who we are.
2,000+ 100+ 124M+ 1.1B 500+
JOURNALISTS NEWSROOMS UNIQUE
MONTHLY
USERS
TOTAL PAGES
PER MONTH
DIGITAL
PRODUCTS
@gregsypolt

3. Happy CustomerReduce Cost
Addressing The Problem With Innovation
Speed
@gregsypolt

4. Evolution of Continuous Integration
Hosted CI Platform
Manual
On-Premise
Self-Service
@gregsypolt

5. The Vision
Shifting LEFT with DevOps and Automated Testing
Code Build Test Deploy MonitorPlanCollaborate
building quality into every stage
Repeat
It takes a COMMUNITY
Dev | Staging | Prod
Discovery Execute Release Review
How We Test Things
Shift RIGHT
@gregsypolt

6. Setting Expectations
@gregsypoltunsplash.com

7. ConfidenceShareAdapt
Expectations for Continuous Integration
Goals
@gregsypolt

8. Ownership
@gregsypoltunsplash.com

9. CI Ownership as a Community
Product EngineeringPlatform Engineering Quality Engineering
</>
</>
Build Unit Integration
Pull Request
Code
√
√
√
Tests
√
√
√
√
√
√
Others Stage Acceptance Others
√
√
√
√
√
√
Tests
Production Others
√
√
√
Synthetics
√
√
√
Tests
Build better
software by
improving
speed,
efficiency, and
collaboration
Teamwork
Vision
Goals
StrategyInnovation Success
MeasureElevate Ideas
Culture of Quality
Best Practices
Collect Requirements
Coach and Mentor
Create Pipeline Code
Maintain Infrastructure
Containerization
Monitoring Results
Identifying Quality Gaps
@gregsypolt

10. Defining A Single Path To Production
@gregsypoltunsplash.com

11. Pre-Commit Commit Acceptance Performance Exploratory Production
Pipeline Stages
Infrastructure
Any plan that defers testing is BROKEN …
Defining a Single Path to Production
@gregsypolt

12. Defining a Continuous Testing Strategy
@gregsypoltunsplash.com

13. Define a Continuous Testing Strategy
Commit
Creation Pull Request
Merged Pull Request
Deployment
Trigger Points
Development
Staging
Production
Environments
Infrastructure
Unit
Module/Component
API
Integration
User Acceptance
Visual
Accessibility
Targeted
Performance
Stress
Security
Scalability
Quality Playbook
Test Continuously
Instant Feedback on Quality
Reduce Risk
Build Confidence
Faster Release Cycles
Shift Left
Disposable Infrastructure
Automate Everything
Application Monitoring
Blue/Green Deployments
Canary
DevOps
@gregsypolt

14. Designing a CI/CD Platform
@gregsypoltunsplash.com

15. The CI/CD Platform
PlatformEng
APIs & Tools
Plumbing
Helm
Source Control
Artifact
Storage
Artifact
Storage
Dev Stage ProdD
CI/CD
Server
CI/CD
Workers
Sauce Labs Applitools SumoLogic
Selenoid
Grid
@gregsypolt

16. Setting CI/CD Standards
@gregsypoltunsplash.com

17. CI/CD Standards
Multiple Teams
Jenkinsfile Notifications
Manage SecretsWebhooksMonitoring More
…
Kubernetes
Containers
Shared Libraries
Dashboards
Artifact Storage Google Storage
@gregsypolt

18. CI/CD Plumbing
Change Logs
Consul
HTTP
Kubectl
Artifactory
DNS
Deployment
Lights Out
Release Notes
Tagging
Vault Credentials
Test Aggregator
GitHub Repo Inform
Semantic Versioning
Build Status Color
JIRA Tickets
@gregsypolt
16

19. Pipeline as Code
Agent
Stages
Stage
Parallel
Basic Pipeline Terms
Restricting Builds
Retrieving Repositories
Docker Image
Build (Compiling Code)
Running Tests
Deploy to Environment
Reporting Test Details
Push Build Artifacts
Notification
Scripting Pipeline Code
In general, most Pipeline scripts look more or less like this…
Jenkinsfile
Commit
Acceptance
Performance
Exploratory
Production
@gregsypolt

20. Pre-Fabricated Pipelines
pipeline {
agent any
environment {
APP_NAME = 'hello'
}
stages {
stage('dependencies') {
when {
expression { opts.stageDeps ==
'true' }
}
steps {
sh "go mod download"
}
}
stage('tests-parallel') {
when {
expression { opts.stageTest ==
'true' }
}
failFast true
parallel {
stage('vet') {
steps {
script {
sh "go vet ${env.APP_NAME}"
}
}
}
stage('unit') {
steps {
script {
sh "go test -v $
{env.APP_NAME}"
...
Pipeline as Code
ci ([
stageDeps: 'true',
stageTest: 'true',
stageBuild: ‘true’,
stageDeploy: ‘true’
])
Pre-fabricated Pipeline
@gregsypolt

21. CI/CD Platform Responsibilities
@gregsypoltunsplash.com

22. Shifting Ownership Responsibilities
27
LEGACY CI/CD SELF-SERVICE
13
HOSTED CI/CD PLATFORM
@gregsypolt

23. By The Numbers
@gregsypolt unsplash.com

24. Availability
LEGACY CI/CD SELF-SERVICE HOSTED CI/CD PLATFORM
Q4 2018
Q1 2019
Q2 2019
99.97%
99.77%
99.89%
@gregsypolt

25. Legacy CI/CD Solution
$14,000/yr
AVERAGE LEGACY CI/CD COST SCALR FARM
LEGACY CI/CD SELF-SERVICE
@gregsypolt

26. Hosted | CI/CD Without Autoscaling
80%COST REDUCTION
$3,000/yr
AVERAGE CI/CD COST W/O AUTOSCALING
HOSTED CI/CD PLATFORM
@gregsypolt

27. 2
4
6
8
10
12
10:30 10:35 10:40 10:45 10:50 10:55 11:00 11:05
Active Build Count (running + queued) Worker Pod Count
BULK BUILD STARTS
BUILDS COMPLETE
COOLDOWN BEGINS COOLDOWN ENDS
Auto-Scaling Workers
@gregsypolt

28. Hosted | CI/CD Platform With Autoscaling
93%COST REDUCTION
$960/yr
AVERAGE CI/CD COST WITH AUTOSCALING
HOSTED CI/CD PLATFORM
@gregsypolt

29. 4,000+AVERAGE BUILDS PER WEEK
600%BUILDS INCREASED
HOSTED CI/CD PLATFORM
Builds | Hosted CI/CD Platform
@gregsypolt

30. 1.08M May 97.9%
1.36M April 97.5%
1.17M March 98.5%
Tests | Hosted CI/CD Platform
HOSTED CI/CD PLATFORM
Total Tests
@gregsypolt

31. The CI/CD Evolution Path
Innovation
Adoption
Maintain
You don’t know your
limits until you push yourself
@gregsypoltunsplash.com

32. Thank You!
Remember why you walked in here today
It’s entirely up to you push your evolution forward
Nothing feels better than new knowledge
@gregsypoltunsplash.com