Damon Edwards (DTO Solutions) presentation at Government Symposium by FedScoop and Red Hat in Pentagon City, VA on Nov 10, 2015

1. DevOps Paradox:
Going Faster Brings Higher Quality,
Lower Costs, & Better Outcomes
© DTO Solutions, Inc.
Nov 10, 2015 ● Pentagon City, VA
Damon Edwards
Managing Partner
DTO Solutions, Inc.

2. @damonedwards
Damon Edwards
damon@dtosolutions.com

3. I get to see a lot…
DevOps Consulting
Operational Improvement
Tools

4. 30x 200x
more frequent
deployments
faster lead
times
60x 168x
the change
success rate
faster mean time to
recover (MTTR)
2x 50%
more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
Recent IT Performance Data is Compelling
High performers compared to their peers…
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report

5. 30x 200x
more frequent
deployments
faster lead
times
60x 168x
the change
success rate
faster mean time to
recover (MTTR)
2x 50%
more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
Recent IT Performance Data is Compelling
High performers compared to their peers…
Faster
Higher
Quality
More
Effective
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report

6. 30x 200x
more frequent
deployments
faster lead
times
60x 168x
the change
success rate
faster mean time to
recover (MTTR)
2x 50%
more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
Recent IT Performance Data is Compelling
High performers compared to their peers…
Faster
Higher
Quality
More
Effective
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report

7. Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report

8. Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report
?!

9. Fast
CheapGood
“Pick Two!”
Conventional Wisdom

10. Fast
CheapGood
“Pick Two!”
Conventional Wisdom

11. Faster, Better, and Cheaper?

12. Faster, Better, and Cheaper?

13. Faster when labor is our largest cost
1
Start 12 mo.
1 111111111 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
Assumptions: Both groups are 30 people at $100/hr; backlogs are identical
Group
A
Group
B
8 “Units of Work”
$6M total cost
$750,000 per unit
24 “Units of Work”
$6M total cost
$250,000 per unit
45 days per “unit”
15 days per “unit”

14. Faster when labor is our largest cost
1
Start 12 mo.
1 111111111 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
Assumptions: Both groups are 30 people at $100/hr; backlogs are identical
Group
A
Group
B
8 “Units of Work”
$6M total cost
$750,000 per unit
24 “Units of Work”
$6M total cost
$250,000 per unit
45 days per “unit”
15 days per “unit”

15. Faster, Better, and Cheaper?

16. Faster, Better, and Cheaper?

17. "In the Lean community, there is a deeply held belief that
lead time is one of the best predictors of quality,
customer satisfaction and employee happiness. And
what we found in our benchmarking work of over 20,000
IT professionals is that it is absolutely true for the
technology value stream, as well.”
Gene Kim

18. "The data shows that throughput and stability metrics
move in tandem — effectively not supporting ITIL claims
that tradeoffs should be made in throughput in order to
get stability. The pattern of needing to trade throughput
for stability simply doesn't appear in the data.”
Nicole Forsgren

19. How about some anecdotal proof?

20. But let’s not talk unicorns…

21. devopsenterprise.ioLet’s talk horses

22. Nordstrom
Courtney
Kissler

23. Nordstrom
Courtney
Kissler
“The DevOps journey began when
Nordstrom stopped optimizing for
cost, and instead started
optimizing for delivery speed.”

24. CSG
Scott
Prugh

25. CSG
Scott
Prugh

26. Why does this work?

27. Why does this work?
Fast Feedback

28. Benefits of focusing on Fast Feedback
• Errors are caught sooner when they are
cheaper to fix (less rework)

29. Benefits of focusing on Fast Feedback
• Errors are caught sooner when they are
cheaper to fix (less rework)

30. Benefits of focusing on Fast Feedback
• Errors are caught sooner when they are
cheaper to fix (less rework)

31. Benefits of focusing on Fast Feedback
• Encourages working in smaller batches (less
simultaneous moving parts.. quick diagnosis and fix)
• Encourages get really good at getting through
the lifecycle (lower the “transaction” cost)
• Errors are caught sooner when they are
cheaper to fix (less rework)

32. Benefits of focusing on Fast Feedback
• Encourages working in smaller batches (less
simultaneous moving parts.. quick diagnosis and fix)
• Encourages get really good at getting through
the lifecycle (lower the “transaction” cost)
• Organization learns faster at all levels (“good at
getting better”)
• Errors are caught sooner when they are
cheaper to fix (less rework)

33. Benefits of focusing on Fast Feedback
• Encourages working in smaller batches (less
simultaneous moving parts.. quick diagnosis and fix)
• Encourages get really good at getting through
the lifecycle (lower the “transaction” cost)
• Organization learns faster at all levels (“good at
getting better”)
• Errors are caught sooner when they are
cheaper to fix (less rework)

34. Benefits of focusing on Fast Feedback
• Encourages working in smaller batches (less
simultaneous moving parts.. quick diagnosis and fix)
• Encourages get really good at getting through
the lifecycle (lower the “transaction” cost)
• Organization learns faster at all levels (“good at
getting better”)
Faster feedback means… higher quality, less
rework, shorter cycle times, and lower costs.
• Errors are caught sooner when they are
cheaper to fix (less rework)

35. But how do we actually do that?

36. Simple System vs Complex System

37. Simple System vs Complex System
•Can be understood by
reducing it to a set of
simpler parts
•Can have complete
information about it

38. Simple System vs Complex System
•Can be understood by
reducing it to a set of
simpler parts
•Can have complete
information about it
•Can perfectly predict
behavior and consequences
of changes

39. Simple System vs Complex System
•Can’t be understood by
reducing it to a set of simpler
understandable parts
•Can’t have complete /
prefect information about it
•Can be understood by
reducing it to a set of
simpler parts
•Can have complete
information about it
•Can perfectly predict
behavior and consequences
of changes

40. Simple System vs Complex System
•Can’t be understood by
reducing it to a set of simpler
understandable parts
•Can’t have complete /
prefect information about it
•Can’t perfectly predict
behavior and consequences
of changes
•Can be understood by
reducing it to a set of
simpler parts
•Can have complete
information about it
•Can perfectly predict
behavior and consequences
of changes

41. Your organization is a complex system2

42. Your organization is a complex system2
Complex
System

43. Your organization is a complex system2
Complex
System
Complex
System
interacting with a

44. Your organization is a complex system2
Complex
System
Complex
System
interacting with a

45. Your organization is a complex system2
Complex
System
Complex
System
interacting with a

46. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning

47. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning

48. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning
Handoff
!
Handoff
!
Handoff
!

49. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning
Application Knowledge
Handoff
!
Handoff
!
Handoff
!

50. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning
Application Knowledge
Operational Knowledge
Handoff
!
Handoff
!
Handoff
!

51. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning
Application Knowledge
Operational Knowledge
Business Intent
Handoff
!
Handoff
!
Handoff
!

52. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning
Application Knowledge
Operational Knowledge
Business Intent
Handoff
!
Handoff
!
Handoff
!
Ownership
but limited
Accountability

53. Silos are the #1 enemy of throughput and quality
Dev Release OpsPlanning
Application Knowledge
Operational Knowledge
Business Intent
Handoff
!
Handoff
!
Handoff
!
Ownership
but limited
Accountability
Accountability
but no
Ownership

54. Your organization is a complex system2
Complex
System
Complex
System
interacting with a

55. Start
Finish
The “Big Bang” Transformation Dream

56. Start
Finish
Fear
Panic
Abort
Maybe
The “Big Bang” Transformation Reality

57. Start
Finish
Fear
Panic
Abort
Maybe
People revert to
legacy behaviors
The “Big Bang” Transformation Reality

58. Start
Finish
Fear
Panic
Abort
Maybe
People revert to
legacy behaviors
“Finish”
● Same as before
● Money/time wasted
● More fear of change
The “Big Bang” Transformation Reality

59. Your organization is a complex system2
Complex
System
Complex
System
interacting with a
Big Bang
“Sliver Bullet”

60. Your organization is a complex system2
Complex
System
Complex
System
interacting with a
Big Bang
“Sliver Bullet”

61. Start
Finish
Start
Finish
“Big Bang” Continuous Improvement
Fear
Panic
Abort
Maybe
Replace the “Big J” with “Little J’s”

62. Your organization is a complex system2
Complex
System
Complex
System
interacting with a
The only way to fix a sufficiently complex
system is to create the conditions for the
system to fix itself.

63. You need an Improvement System

64. You need an Improvement System
● Not for your technology
● Not to build individual skills

65. You need an Improvement System
● Not for your technology
● Not to build individual skills
● … but for how your organization works

66. You need an Improvement System
● Not for your technology
● Not to build individual skills
● … but for how your organization works

67. You need an Improvement System
● Not for your technology
● Not to build individual skills
● … but for how your organization works
in the context of

68. Make it Repeatable and Sustainable
Improvement System is going to have to…

69. Make it Repeatable and Sustainable
•Scale quickly
•Span multiple organizational boundaries
Improvement System is going to have to…

70. Make it Repeatable and Sustainable
•Scale quickly
•Span multiple organizational boundaries
•Keep improvement efforts aligned
Improvement System is going to have to…

71. Make it Repeatable and Sustainable
•Scale quickly
•Span multiple organizational boundaries
•Keep improvement efforts aligned
•Work with substantial numbers of legacy
technologies
Improvement System is going to have to…

72. Make it Repeatable and Sustainable
•Scale quickly
•Span multiple organizational boundaries
•Keep improvement efforts aligned
•Work with substantial numbers of legacy
technologies
•Develop existing staff in mass
Improvement System is going to have to…

73. Make it Repeatable and Sustainable
•Scale quickly
•Span multiple organizational boundaries
•Keep improvement efforts aligned
•Work with substantial numbers of legacy
technologies
•Develop existing staff in mass
•Be self-funding after initial seed investment
Improvement System is going to have to…

74. Example: “DevOps Kaizen”
Value Stream Mapping
Retrospectives / Planning
Service Delivery
Metrics
Systems Thinking
Improvement
Kata

75. Provides a mechanism to introduce new ideas
Improvement System
DevOps Kaizen
Design Patterns

76. Examples of DevOps Patterns
Continuous Delivery
Everything in SDLCTesting in Production
Infrastructure as Code Blameless Post Mortems
Service Aligned Org
Ops as a Service ChatOps Prod-like Dev Environments

77. Recommended Reading

78. Want to talk Improvement Systems or DevOps?
@damonedwards
damon@dtosolutions.com
Federal Agencies can also contact:
Peter O'Donoghue
podonoghue@csgov.com