Brian Paulsmeyer, a Sr. Architect at Centric St. Louis, spoke about DevOps on September 29th at Agile Gravy Conference in St. Louis. Here's his presentation, which starts with Agile development pitfalls that plague teams, moves into the actual capabilities that a team requires to be successful, and finally describes concrete implementations to achieve “Done Means Done” development.

1. Sprinkle Dev, Sprinkle Ops, lets make Cake not Mud Pies
Brian Paulsmeyer
Agile Gravy: September 29, 2016

2. Agile Development Pitfalls
… that warn us of mistakes in our agile implementation
10/4/2016
www.centricconsulting.com 1

3. Waterfall in 2 Week Iterations
•No ordering of stories
•Gold plating of features
•“Agile Waterfall” Assumptions
• Learning from usage of system won’t change features
• “All features will come together” in last day of last sprint
• “Pause the world for 3+ years for us to finish”
• “Value Proposition: All or Nothing”
• Architecture can be “known” and not change
• Deployments are still hard
•Unfortunate ending for “Agile Waterfall”
• “We ran out of money, and have nothing in production.”
10/4/2016www.centricconsulting.com 2

4. Constant Redefining of “Definition of Done”
•Done should simply mean done
•Burndown/Burnup no longer reliable
•Feature inversion occurs when time spent on defects and maintenance
overtakes feature work
10/4/2016www.centricconsulting.com 3
20 20 20
18 17 17 16 16 15
13
0 0 0
2 3 3 4 4 5
7
1 2 3 4 5 6 7 8 9 10
FEATURE INVERSION
Features Defects and Hidden Work

5. Only focusing on Unit Tests and Manual Testing
•Unit tests
• Won’t ensure business workflows
• Only allows minor refactoring
• Failure: Mars Climate Orbiter (1999)
• Team A: Pounds for unit of force
• Team B: Newtons for unit of force (1 lb = 4.45 N)
• Disintegrated in Mars atmosphere!
• Each “Unit”/Component worked in isolation
10/4/2016www.centricconsulting.com 4

6. Continuous Delivery and DevOps
… simple concept, requires diligent implementation
10/4/2016
www.centricconsulting.com 5

7. No more “Throw It Over the Wall”
10/4/2016www.centricconsulting.com 6

8. DevOps enables Continuous Delivery
•Team performs activities across disciplines
•Apply principles from iteration 0 through delivery
10/4/2016www.centricconsulting.com 7
Regression
UAT
Integration
Testing
DevOps – Full Lifecycle Delivery Pipeline and Container Management
Local
Environments
Builds
Unit Tests
Merge
Procedures
Log
Interpretation
Dependency
Mgmt
Dev
Deployments
Traditional DEV activities Traditional OPS activities Traditional QA activities
Prod
Deployments
Performance
Mgmt
Monitoring
QA
Deployments
Branch Mgmt
Change Mgmt
Log
Management
Configurations
Integration
Deployments
Performance
testing

9. What It Takes
Whether it be security, regulatory, software limitations, etc., there are
many reasons why organizations say CD is impossible
10/4/2016www.centricconsulting.com 8
Tooling
Process
Culture
Architecture

10. Basic Definitions and Levels of Maturity
Continuous…
10/4/2016www.centricconsulting.com 9
Automated Deployment to ProductionDeployment
Manual Compile/Build, Manual Testing, Manual DeploymentPain
Automated Deployment to Integration/Staging/Pre-Prod
Deployment Toolset Constant across all Environments
Delivery Automated Integration/Workflow Tests
Automated Component Builds
Automated Component Tests*
Integration Automated Unit Tests
* Many teams mistakenly consider Continuous Integration (CI) complete with only automated unit tests

11. Pipeline
•Recipe for building, deploying, and testing the system
•Grouping of discrete stages for a complete workflow
•Coordinates disparate technologies
•Provides feedback of system state across environments
10/4/2016www.centricconsulting.com 10
Business &
Customer
Develop
Customer
Deploy
to
Production
Deploy
to
Staging
Deploy
to
QA
Build
and Unit Test
Automated
Integration /
Workflow
Testing

12. Detailed Pipeline
10/4/2016www.centricconsulting.com 11
Business &
Customer
Backlog
Consolidate
Features
and Hotfixes
Iteration
Planning
Build
Server“X”
Stories
Development
1 Active Story
Per Developer
1
Commit
Acceptance
Test
Customer
Build
Artifact
Acceptance Test
Environment
QA
Testing
QA
Environment
Auto-Trigger Pipeline:
Installs To AIT Environment
Executes Smoke and
Acceptance Tests (<10 min)
Manual Trigger Pipeline:
Installs To
QA Environment
Executes Smoke Tests
UAT
Testing
Manual Trigger
Pipeline:
UAT Environment
UAT Smoke &
UAT Tests
Production
Manual Trigger
Pipeline:
Prod Environment
Prod Smoke
Tests
UAT
Environment

13. Product Pipeline
10/4/2016www.centricconsulting.com 12

14. Pipeline Workflow Tools
… to speed implementation
10/4/2016
www.centricconsulting.com 13

15. Pipeline Stage Implementation
•Pipeline workflow steps trigger external tools
10/4/2016www.centricconsulting.com 14
Business &
Customer
Develop
Customer
Deploy
to
Production
Deploy
to
Staging
Deploy
to
QA
Build
and Unit Test
Java
.Net
TypeScript
PHP
etc.
Ant
Maven
MSBuild
Make
etc.
JUnit
Nunit
etc.
Bash
PowerShell
Chef/Puppet
VMs
Docker/CoreOS
etc.
Automated
Integration /
Workflow
Testing
Selenium
Cucumber
JBehave
SpecFlow
Codeception
etc.
Same Tools
as Deploy to QA
Only Configuration
Differences
Same Tools
as Deploy to QA
Only Configuration
Differences
Build Tool Jobs (ex. Jenkins)

16. Key Pipeline Coordination Tool Features
•Pipeline Triggers
• Automatic on code check-ins to source control
• Manual
• Programmatic API
•Workflow
• Parallel steps
• Sequential steps
• Failure-only steps
• Graphical view of steps
•Artifact Tracking
• Traceability from check-in to final deployment
• View of deployed version in each environment
• History of deployments
• Ability to trigger updates to agile tracking tools (ex. JIRA)
•Environment Support
• Same deploy scripts, allows configuration changes per environment
10/4/2016www.centricconsulting.com 15

17. Pipeline Workflow Tool : Jenkins
•On-Site
•Version 2.0 Provides Workflow View
•Software Language Agnostic
10/4/2016www.centricconsulting.com 16

18. Pipeline Workflow Tool : ThoughtWorks Go
•On-Site
•Software Language Agnostic
10/4/2016www.centricconsulting.com 17

19. Pipeline Workflow Tool : Octopus Deploy
•.Net
10/4/2016www.centricconsulting.com 18

20. Automated Testing
… so every day can be deployment day
10/4/2016
www.centricconsulting.com 19

21. Automated Component/Integration/Workflow Testing
•Opinions of why automated testing isn’t needed
• “Manual testing is sufficient”
• “Business only wants to spend money on features”
• “Developers only want to code features”
• “Our <insert technology> is too hard to test with automation”
• “Automated tests run too slow”
10/4/2016www.centricconsulting.com 20
Best Case: Missed Deadline
Worst Case: Catastrophic Failure in Production
Result of skipping
automated tests

22. Selenium WebDriver
Direct Test using Selenium WebDriver
WebDriver driver = new FirefoxDriver();
// Navigate to webpage
driver.get("http://www.google.com");
// Find the Search Field (UI Locator)
WebElement element = driver.findElement(By.name(“Search"));
// Fill in the data
element.sendKeys(“something interesting");
// Now submit the form. WebDriver will find the form that contains the element
element.submit();
// Assert the test passed
10/4/2016www.centricconsulting.com 21
Selenium
WebDriver
Web Page
Test
Test
Test
Test

23. Page Object Model
10/4/2016www.centricconsulting.com 22
• Separate Test Code and UI Locators
• Supports Don’t Repeat Yourself (DRY) for Element Locators
• Less Technical Test Case
SearchPage page = new SearchPage(selenium);
// Enter Search information
page.search(“something interesting");
// Now submit the form
page.submit();
// Assert the test passed
Page Object
Model(s)
Web Page
Test
Test
Test
Test
Selenium
WebDriver
ServerAPI

24. BDD Testing
• Gherkin (English) Text File
• Step Definitions Map Between English Test and System
• Business Readable Test Case
Given user searches for “something interesting”
When user performs the Search
Then assert something is true
10/4/2016www.centricconsulting.com 23
Page Object
Model(s)
Web Page
Step
Definition
Step
Definition
Step
Definition
Selenium
WebDriver
ServerAPITest
Test
Test
Test

25. Capabilities
… we need to be successful
10/4/2016
www.centricconsulting.com 24

26. Rapid Response to Changing Business
•Need to be able to react to disruptive technology and business models
•Stagnant architecture limits new technology adoption
• Changes blocked by fear of breaking system
• Inability to deploy new systems to production
• Unable to adopt new security upgrades
• Unable to adopt new software versions
10/4/2016www.centricconsulting.com 25

27. Drive Defects From Most Used Features
“You can’t test quality in”
At a minimum make sure defects aren’t in the standard user workflows
10/4/2016www.centricconsulting.com 26

28. Adjustable/Changeable Architecture
•Magic Quadrant Architecture
• “Best” choice today ≠ Best choice tomorrow
• Failed technologies and companies in all parts of magic quadrant
•Natural product End-of-Life requires new architecture
•Continuous Delivery and DevOps allows fluid architecture changes
10/4/2016www.centricconsulting.com 27
(Object-Oriented Database)

29. Cadence
•What if everyday was a normal day
•What if those normal days were release days
•What if those release days were multiple releases throughout the day
•Low Risk On-Demand Releases
10/4/2016www.centricconsulting.com 28

30. Deployment Styles
… for production deployment
10/4/2016
www.centricconsulting.com 29

31. Standard Deployment
•Deployment Steps
• Disable user access
• Upgrade servers from V1 to V2
• Test deployment
• Enable user access
• Upgrade destabilizes running environment
•Problem Mitigation
• Re-Enable backup if possible
• Developer troubleshooting requires
customer downtime
10/4/2016www.centricconsulting.com 30

32. Blue Green
Router /
Load Balancer
Blue-Green Deployment
•Two Identical Production Environments
• Blue
• Green
•One Active: Router Controlled
•Deployment Steps (Green Active)
• Deploy to Blue environment – No users
affected
• Perform smoke tests on Blue environment
• Optional: Switch subset of users to Blue
(similar to Canary releases)
• Switch all users to Blue environment
•Problem Mitigation
• Don’t switch to 2nd environment
• Developer troubleshooting has no customer
impact
10/4/2016www.centricconsulting.com 31

33. Canary
Router /
Load Balancer
Canary Deployment
•One Production Environment
•Deployment Steps
• Deploy to 1 server (Canary)
Optional: Provision and build new server from
scratch
• Router sends portion of users to Canary
• Random
• Internal users only
• User-type based (ex. Test users)
• Gradually upgrade servers to V2
• Gradually switch users to V2 servers
•Problem Mitigation
• Don’t switch additional users to Canary
environment
• Developer troubleshooting has no customer
impact
10/4/2016www.centricconsulting.com 32

34. Case Studies
… that show it can be done

35. Before Build Automation
•23 Teams
•  150 Team Members (Developers, Quality Engineers, Scrum Masters)
•  100 Developers
•Service Oriented Architecture
• Managers, Engines, Data Access
•Baseline End of PI6 (SAFe)
• PI6 demo system setup
• Two weeks to setup
• Manual Installations – not repeatable
• Individual Teams created “magic instructions” in isolation to install and run solution
• Some teams unable to run full solution
• Custom data loaders used to skip workflow steps
• Developers unable to test full workflow
• Movement of data through workflow steps untested till end of PI
• Update of component versions caused “Day of Downtime” for developers to fix unexpected
issues
10/4/2016www.centricconsulting.com 34

36. After Build Automation: Jenkins Easy Button Builds
• Starting Point: a machine with Windows — that’s it!
• The rest is automated:
− Provisioning: verify/install tools takes ~50 minutes
− Puppet Agent, PowerShell, 7-Zip, VNC
− SQL Server, .Net, Visual Studio
− Noticia Repono Client
− Install Custom Software takes 5-10 minutes
− Clean system from any previous runs
− Install/Configure Base Components
− Install System
− Install and Run Smoke Tests
At this point the system is ready to be used to do further testing, demos, etc.
• Deploy cycle repeats
ChunkyMonkey
Vanilla

37. Definition of Done
… that makes our customers happy
10/4/2016
www.centricconsulting.com 36

38. Continuous Delivery Definition of Done
•Automated Deployments
• Fully Automatic or Push-Button Deploys
• Same Executable Scripts for all Environments
• No Deployment Hero
•Automated Testing
• Unit
• Integration
• Full System
•Always Shippable
• No Temporary Code
• No Untested Code
• Feature Toggles
•One Backlog
• Production Bug Fixes
• New Features
10/4/2016www.centricconsulting.com 37

39. About Me
10/4/2016www.centricconsulting.com 38
• Our highest priority is building lifelong
relationships with clients based on trust,
respect, and collaboration.
• We invest in our talented team and
support their well being by keeping them
challenged and inspired.
• Our localized company structure allows
us to play very active roles in the lives
of our families and community.
This relationship-centric focus keeps us passionate, committed and
motivated in all facets of our lives. That’s why our team is here to stay.
Centric Consulting
• 17 years as a software developer and architect
• Full stack developer in Java, .Net, TypeScript (Angular 2), and PHP
• DevOps tooling experience including VMs, Docker, Chef, and Puppet
• Continuous Delivery and Automated Testing implementations
• Experience across industries including FDA and SEC regulated
Continue the Journey • http://centricconsulting.com/centric-st-louis-scaled-agile-workshop/
Thank You for Attending!