Tech Talk: The Continuous Delivery Metrics That Will Make Your Pipeline Fast and Your Apps Better

1. World®
’16
Tech Talk: The Continuous Delivery Metrics
That Will Make Your Pipeline Fast and Your
Apps Better
Dan Zentgraf – Sr. Principal Consultant – CA Technologies
DO5T20T
DEVOPS

2. 2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

3. 3 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Abstract
A key principle of continuous delivery is continuously optimizing your
application delivery pipeline for quality and speed. This relies on a deep
understanding of the varied processes at different stages of your pipeline
and their potential to kill delivery times and productivity.
In this session, learn why you need both results and performance metrics to
uncover deep insights that drive substantial long term performance
increases, not just incremental change. Understand how applying metrics
throughout the pipeline fundamentally enables feedback loops across your
continuous delivery toolchain. Explore the integral role of CA Release
Automation Continuous Delivery Edition in tracking and applying software
development lifecycle metrics to uniquely optimize your processes and
environments.
Dan Zentgraf
CA Technologies
Sr. Principal Consultant,
Presales

4. 4 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Topics for Today
INTRODUCTION
METRICS
RELATED CA WORLD SESSIONS
INSTRUMENTATION
VALUE STREAM MAPPING CONCEPTS
KEY MEASUREMENT ITEMS
1
2
3
4
5
6

5. 5 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Core Values of DevOps Movement – “CAMS”
CultureC
A
M
S
Automation
Measurement
Sharing

6. 6 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Automation Is Awesome
THERE ARE, HOWEVER, A FEW PRE-REQUISITES
TO A SUCCESSFUL EFFORT
§ Where do we start?
§ How do we know we are making progress?
§ How do we know we are getting value from the
effort? Are we doing the right things?
Let’s Automate ALL THE THINGS!!!

7. 7 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
WE
“If You Can't Measure It,
You Can't Improve It. ”
–Peter Drucker
“I never guess. It is a capital mistake
to theorize before one has data.
Insensibly one begins to twist facts to
suit theories, instead of theories to
suit facts.” – Sir. Arthur Conan Doyle
“If the statistics are boring,
you’ve got the wrong
numbers.” –Edward Tufte
“There are lies, damn lies, and
statistics.”
–Mark Twain

8. 8 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Results Metrics and Performance Metrics
Myth: Result metrics will improve performance
Reality: You need both & they must be aligned
RESULT METRICS
Show the RESULT
of an Action or Process
PERFORMANCE METRICS
Track and Measure the Actions and
Processes that LEAD TO the Result

9. 9 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Continuous Delivery Metrics
THE CD CLASSIC: “Cycle Time”
Actual work time on an item in a
process
DEVOPS: “Lead Time”
Elapsed time from when request
was made to delivery
Cycle Time and Lead Time
are RESULT metrics
PERFORMANCE metrics are less well defined
§ Much variance based on individual team or
business situation
§ Organizational barriers create challenges

10. 10 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Where / What to Instrument
VALUE STREAM MAPPING
A lean manufacturing or lean enterprise technique used to document, analyze and
improve the flow of information or materials required to produce a product or service
for a customer.
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time

11. 11 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Enterprise Delivery Processes Are Complex…
Any given environment has multiple involved streams / sub processes
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
Validate
Deploy
Configure
Provision
PROMOTION DECISION
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time

12. 12 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
And They Multiply Through the Pipeline…
Dev/Int QA Prod
Requires visibility to second level to understand actual mix of Cycle / Wait time to
find bottlenecks.
§ Need the core automation driver + oversight in order to track
manual processes with the automation across the whole pipeline
§ THIS is the metrics engineering all of us must do!
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
Validate
Deploy
Configure
Provision
PROMOTION DECISION
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
Validate
Deploy
Configure
Provision
PROMOTION DECISION
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
Validate
Deploy
Configure
Provision
PROMOTION DECISION
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time

13. 13 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Key Operational Metrics at Each Pipeline Phase
How long till you have a deployable
package?
Timeframes for key Tasks per
environment
§ How long to provision / configure /
deploy / validate?
§ Failure rate at each step
Reduction in manual Tasks
Reduction in maintenance of
automation
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time
Validate
Deploy
Configure
Provision
PROMOTION DECISION
STEP STEP STEP STEP
Cycle Time Cycle Time Cycle Time Cycle Time
Lead Time
Wait
Time
Wait
Time
Wait
Time

14. 14 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Key Operational Metrics for Quality in the Flow
Code quality metrics
Feature completeness
§ How far does a change go before being kicked out of process
§ Rate of feedback-based rework
Delivery quality
§ “Smoke Test” passage rate per environment
§ Decrease in unintended breakage / broken dependency events
Application Health / Usage
§ Monitoring system / reliability
§ App feature tracking
You can only BUILD quality in. You cannot test quality in.
Dev
QA
Deploy
Run

15. 15 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Pipeline Delivers Statistics to Release Management
Discipline-specific AND Overall Flow Examples
Agile Metrics
§ Velocity
§ Burn down / burn up
§ Lead time
Requirements
& Defects
Concurrent
Testing
Dev &
Build
Pre-
production
Production
& Monitoring
CA Release
Automation +
Continuous
Delivery Edition
Agile
Requirements
Designer
Service
Virtualization
Test Data
Management
Code Quality
§ Coverage
§ Unapproved Open Source
§ AppSec Remediation
§ Cycle Time
Testing
§ Dependency breakage rate
§ Stage time per phase
§ Smoke test passage
§ Cycle Time
Production Metrics
§ MTTR
§ Support incident rate
§ Usage data
UAT
§ Rejection / Rework rate
§ Stage time per phase
§ Smoke test passage
§ Cycle Time

16. 16 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Applying Metrics -> Analytics
Ideas from The Lab – KPI Dashboard / Report Card

17. 17 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Your Pipeline Metric Goals Are…
§ Understand how your Performance and Result Metrics relate to each other
§ Analyze your Value Stream
§ Identify your Instrumentation points
§ Track both Efficiency and Quality metrics
§ Listen to your metrics and apply the lessons
Summary
A Few Words to Review

18. 18 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Recommended Sessions
SESSION # TITLE DATE/TIME
DO5X07S
Jeweler’s Mutual Insurance Streamlines Complex App
Deploy w/ CA RA
11/17/2016 at 12:45 pm
DO5X12S
EchoStar Transforms Complex Software Releases to
Customer Network Set Top Boxes with CA RA CDE
11/17/2016 at 01:45 pm
DO5T19S
GM Financial Builds a Sustainable, Holistic, Continuous
Delivery Practice
11/17/2016 at 04:30 pm

19. 19 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Must See Demos
Modernize
App Delivery
Integrated CD
Theater 5 - DOV501P
Deliver Test
Data Faster
Test Data Manager
Theater 5 - DOV511P
Deliver Better
Apps
Service Virtualization
Theater 5 - DOV507P
Orchestrate
Your Release
Release Automation
Theater 5 - DOV513P

20. 20 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Stay connected at communities.ca.com
Thank you.

21. 21 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
DevOps – Continuous Delivery
For more information on DevOps – Continuous Delivery, please
visit: http://cainc.to/PiTFpu