20160517 Lean Kanban North America 2016 Conf Bringing DevOps to an Entrenched Legacy Environment with Kanban

Craeg Strong
CTO, Ariel Partners
May 17, 2015
San Diego, CA
© Copyright Ariel Partners 2016 *sales@arielpartners.com ((646) 467-7394 1
Bringing DevOps to an
Entrenched Legacy Environment
With Kanban

Software Development since 1988
Large Commercial & Government Projects
Agile Coach / (Recent) Kanban Trainer
Performance & Scalability Architect
Apache Ant Open-Source Contributor
New York & Washington DC Area
CTO, Ariel Partners
AKT, CSM, CSP, CSD, CSPO, PSM,
PMI-ACP, PMP, LeSS, SAFe
www.arielpartners.com
cstrong@arielpartners.com
@ckstrong1
Craeg Strong
© Copyright Ariel Partners 2016 *sales@arielpartners.com ((646) 467-7394

Agenda
Define: DevOps
The Project
The Case For DevOps
Implementing DevOps For Legacy
q Build Automation
q Test Automation
q Deployment Automation
q Configuration As Code
Recap and Next Steps

DEFINE: DEVOPS
4© Copyright Ariel Partners 2016 *sales@arielpartners.com ((646) 467-7394

What Is DevOps?
5
DevOps (a clipped compound of "development" and
"operations") is a culture, movement or practice
that emphasizes the collaboration and
communication of both software developers and
other information-technology (IT) professionals
while automating the process of software delivery
and infrastructure changes.
DevOps - Wikipedia, the free encyclopedia
https://en.wikipedia.org/wiki/DevOps

DevOps In The Large
6
•Shared Templates & Frameworks
•Standards are not Static, Evolve Based on Team Feedback
•Multiple Tool chains: Java, .NET, MEAN
Tools
•Emphasis on Collaboration/Trust
•Encourage New Ideas, Limit Downside via Fail-Fast
•Blameless Post-Mortems
•Servant Leadership
•Efficiency vs Thoroughness Trade-off (ETTO Principle)
Culture
•Cross Training
•Emphasis on Continuous Learning
•Transition To Management Not Required for Career
Advancement
•Lightweight, Adaptable
•Maximize Throughput, Not Utilization
•Tight Feedback Loops
•Continuous Service Improvement
ProcessOrganization

Aspects I Will Focus On
7
Build
Automation
Test
Automation
Deployment
Automation
Configuration
As Code

THE PROJECT

Criminal Justice Program

System Evolved Over Many Years
10
1990
Pilot
DNA
Identification
Act
1994 1998
Client-Server: VB6
SQL
Server
2002 2012
C# VB.Net
Multi-Language SOA Re-Architecture
VB6
2008
2M Lines
of Code

Change Control
Board Meeting
Traditional Waterfall Lifecycle
M5 M6 M7 M8 M9 M10 M11 M12
Year One
M1 M2 M3 M4
Year Two
M1 M2 M3 M4
Require-
ments
Design
Code And Unit Test (CUT)
System
Test
GAT
UAT
PRR
PDR
CDR
TRR
TRR
OAR

Federal Contracting Environment
BASE YEAR
8/29/2012 – 8/28/2013
2013 2014 20152012
OPTION YEAR 1
8/29/2013 – 8/28/2014
. . .
Initial Process Flexibility
High Profile Agile Success (Sentinel)
Agile Support In Federal Acquisition Regulations
High Cost Of Process Change
100% Outsourcing Model
Zero Access to Larger Organization

Challenges Inherent to Legacy
Environments
13
q Mission-Critical
q Heavily Regulated
q Missing or Out of Date
q Significant Documentation Requirements
q Mostly Manual
q System Not Designed For Testability
q Resistant to Change
q Battle Fatigue
q Optimized For Non-Collaboration
q May Inhibit Pair Programming
q Heavyweight, Prescriptive
q Manually-Intensive Reporting
q Lengthy Approval Process
q Skeptical of Open Source
System
Criticality
Documentation
Testing
Staff
Facility
Governance
Procurement

Challenges Implementing Agile
Practices In A Legacy Environment
14
q Training, Documentation Assumes New (“Greenfield”)
Project
q Relatively Little Guidance Regarding Legacy
(“Brownfield”) Projects
q How Can We Release To Production Every Two Weeks
Given A 6-Week Approval Cycle?
q How Can We Adopt Two Week Sprints When Regression
Testing Consumes The Entire Team For Eight Weeks?
q Can’t Fix Everything At Once
Agile Training &
Literature
Agile Practices
q Automation Tools Poorly Supported By Older Platforms
q Legacy Tends To Be Data-Heavy, Little Tool Support for
Data Validation
q Business Logic Tends To Be In GUI and Data Layers,
Rendering Traditional Unit Testing Useless
Agile Toolset

Why Would Anyone
Sign Up For This?

The Mission

The Challenge

The Team

THE CASE FOR DEVOPS

Systems Approach To Implementing
Kanban (STATIK)
20
1. Understand what makes the service “fit for
purpose”
2. Understand sources of dissatisfaction
regarding current delivery
3. Analyze sources of and nature of demand
4. Analyze current delivery capability
5. Model the service delivery workflow
6. Identify & define classes of service
7. Design the kanban system
8. Socialize design & negotiate implementation

Demand Analysis
Work Item Type Class Of Service
Business User Story Standard
Infrastructure User Story Standard
?
Tier III / Triage Expedited
New Forensic Lab Support Scripts Fixed Date
Hot Fix Expedited
Escaped Defect Expedited
Documentation Deliverable Fixed Date
Documentation Rework Expedited
Technical Chore Intangible

Scrum Tool Setup

Delivery Capability Analysis
0
5
10
15
20
25
Sprint 5: User Story Completion Curve
Testers
Waiting For
Code
Limited Time
For Testing

Initial Workflow
Longer Sprint
With Bigger
Batch-Size
Bottleneck
Lots of Blocked
Issues, Defects, and
Rejected Fixes

Sources of Dissatisfaction
q Developers Check-in In Code That Breaks the Build
q High Concentration of Defects: Fixing One Issue
Breaks Two More
q Automated Tests Too Difficult To Write
q Deployment To Test Environments Lengthy and
Error-Prone
q Test Set-Up Lengthy and Error-Prone, Tear-Down
Usually Skipped
q Many False Positive Defects Due To Configuration
Issues

Analyzed Workflow

BUILD AUTOMATION

Build Machine
Typical Build Automation
28
TFS
TeamCity
Binary Repository
Compile Code Run Tests Package Publish
Compile Code
Branch B
Run Tests
Branch B
Package
Branch B
Publish
Branch B
Compile Code
Branch C
Run Tests
Branch C
Package
Branch C
Publish
Branch C

Build Machine A
Build Automation For
Legacy System
29
TFS
TeamCity
Binary Repository
Compile Code
Run Unit Tests
Package Publish
Run SQL Tests
Drivers, Patches
Build Machine B
Compile Code
Branch B
Run Unit Tests
Branch B Package
Branch B
Publish
Branch B
Run Unit Tests
Branch B

Updated Workflow
“Build Ready”
Step Has Been
EliminatedBut We Still Have
Significant
Quality Problems

TEST AUTOMATION

Typical Test Automation
32
GUI
Presentation Logic
Server
Business Logic
DB
SQL (CRUD)
Test
Start
Test
End
üMinimal Setup Required
üCode-Only Tests
üVery Fast Test Execution
üNo Side Effects

Test Automation
For Legacy Systems
33
GUI
Presentation-Side
Business Logic
Server
Data Transfer
DB
SQL
Business Logic
üSignificant Setup Required
üTests Require Database
üSlow Test Execution
üSignificant Side Effects
Test
Start
Test
End

Build Machine A
Incorporating Test Automation
Into The DevOps Pipeline
34
TFS
TeamCity
Binary Repository
Compile
Code Run Unit
Tests
Package Publish
Run SQL
Tests
Drivers, Patches
Prepare
Test DB
Run Tests
Batch 1
Build Machine B
Prepare
Test DB
Run Tests
Batch 2
Build Machine C
Prepare
Test DB
Run Tests
Batch 3
Test DB Image

Updated Workflow
But It Still Takes
Too Long Before
We Can Test
Quality Is
Slowly Starting
To Improve

DEPLOYMENT AUTOMATION

Typical Deployment Automation
Pipeline
37
TFS
TeamCity
Version Control
Server Build / CI
Server
Binary
Repository
Developer
Deployment
Service
Tester

Deployment Process
Deployment Automation For Legacy
System
38
Deployment
Service
Tester
TFS
Developer
PreDeploy.ps1
StopServers
UninstallPreviousVersion
DeleteOldDatabase
Deploy.ps1
RestoreTestDB
InstallSoftware
PostDeploy.ps1
AddTestUsers
ConfigureSettings
CleanupPreviousSession

Updated Workflow
But We Are Still
Getting False
Positive Defects
“Test Setup”
Step Has Been
Eliminated
“Deploy” Step
Has Been
Streamlined

CONFIGURATION AS CODE

Deployments Have Two Pieces:
Application Code and Operating Platform
41
Operating Platform: Configuration
Application: Code
Database
Matching Engine
Application Server
Graphical User Interface

Configuration As Code:
Typical Implementation
Web Server Compute Server Database Server
Configure
TFS
Configurations

Configuring The Operating Platform
For A Legacy System (1 of 2)
No Step Where
1 OS Install
2 Set Hostname & IP Address
3 Set Time zone
4 Install Active Directory Services Production Only
5 Activate Windows
6 Create Messaging & Temp Folders
7 Configure File Share
8 Set Folder Permissions
9 Set Environment Variables
10 Disable Windows Firewall
11 Enable Remote Desktop Dev/Test Only
12 Configure Active Directory Groups

Configuring The Operating Platform
For A Legacy System (2 of 2)
No Step Where
13 Install 7-Zip Dev/Test Only
14 Install Octopus Tentacle Dev/Test Only
15 Configure Octopus Tentacle Dev/Test Only
16 Register New Tentacle With Octopus Server Dev/Test Only
17 Install SQL SMO and CLR Packages
18 Install MSMQ
19 Install .NET 3.5
20 Install SQL Server
21 Install IIS
22 Configure FTP/FTPS
23 Configure PKI Certificates
24 Install hMailServer Dev/Test Only,
Single Machine

Evolution Of The Process
STEP ONE
100% Manual
300 Page Document
STEP THREE
The Configurator
STEP TWO
Partially Automated
Individual Scripts

Enter: The Configurator
File Shares
Windows SXS
Files
Third Party
Installer Files
Test Environment
Nodes
Targeting
File
Configurator
Script
Configure
DevOps Engineer

Updated Workflow
Greatly Reduced
Incidence of False
Positive Defects
Configuring New Environment Reduced
Weeks à Hours
Validating Existing Environment Reduced
Days à Minutes

Key Takeaways
48
1. What DevOps Practices Can Benefit My Project?
qBuild Automation
qTest Automation
qDeployment Automation
qConfiguration As Code
2. DevOps for Legacy
qChallenging, Yet Transformative
3. Kanban System and STATIK
q Lightweight and Effective Guides

Next Steps
49
qWhere Are The Metrics?
qDeepen Kanban Implementation
qLeverage Upcoming Punctuation Point
qEstablish Real WIP Limits
qTimebox à Service Level Agreements
qInvestigate Improved Tool Support

20160517 Lean Kanban North America 2016 Conf Bringing DevOps to an Entrenched Legacy Environment with Kanban

Recommended

Recommended

More Related Content

Similar to 20160517 Lean Kanban North America 2016 Conf Bringing DevOps to an Entrenched Legacy Environment with Kanban

Similar to 20160517 Lean Kanban North America 2016 Conf Bringing DevOps to an Entrenched Legacy Environment with Kanban (20)

More from Craeg Strong

More from Craeg Strong (20)

Recently uploaded

Recently uploaded (20)

20160517 Lean Kanban North America 2016 Conf Bringing DevOps to an Entrenched Legacy Environment with Kanban