The document discusses choosing enterprise tools to build a continuous delivery toolscape. It covers defining the scope of such a toolscape beyond development teams, understanding the different categories of continuous delivery tools, and criteria for evaluating products for enterprise use. It provides three real-world examples of toolscapes from different companies and key points about each. The presentation aims to help organizations standardize on tools as DevOps initiatives scale up across the enterprise.

1. How to Choose EnterpriseTools to
Build OutYour Continuous Delivery
Toolscape
Andrew Phillips, VP Products

2. 2 Copyright 2014.
About Me
▪ VP Products for XebiaLabs
▪ Lots of enterprise software development on
high-performance systems
▪ Been on both sides of the “Dev…Ops” fence
▪ Active open source contributor and committer
▪ Regular meetup, conference etc. presenter

3. 3 Copyright 2014.
About Me

4. 4 Copyright 2014.
Agenda
▪ Continuous Delivery, DevOps & Tooling
▪ The Scope of an Enterprise CD Toolscape
▪ Understanding Continuous Delivery Tools
▪ Product Evaluation Criteria for Enterprise Use
▪ 3 Real-world Examples

5. 5 Copyright 2014.
About XebiaLabs
▪ We build software to support Continuous Delivery at scale
▪ We know how to implement CD “in the real world”

6. 6 Copyright 2014.
About XebiaLabs
Application Lifecycle
Management
Change and Release
Management
CI Servers Test Tools
Middleware
Provisioning Tools
Cloud Platforms
Continuous
Delivery
Management
Automated
Application
Deployments
Test
Management
& Analytics –
“quality hub”

7. 7 Copyright 2014.
Continuous Delivery, DevOps &Tooling
▪ It’s been said many times: “Continuous Delivery & DevOps >> Tooling”
▪ In practice, the majority of CD & DevOps initiatives are automation initiatives
− Often, localized activity driven by individual teams or Tech Heroes
▪ Automation addresses many of the immediate pain points felt by teams
“We spend ages waiting for
a test environment to become
available, and further time
trying to undo all the changes
made be the previous team”

8. 8 Copyright 2014.
Continuous Delivery, DevOps &Tooling
▪ It’s been said many times: “Continuous Delivery & DevOps >> Tooling”
▪ In practice, the majority of CD & DevOps initiatives are automation initiatives
− Often, localized activity driven by individual teams or Tech Heroes
▪ Automation addresses many of the immediate pain points felt by teams
“Many of the problems in
our code only show up in
pre-production, because the
developers can’t put together
a local environment that’s
sufficiently production-like”
“We spend ages waiting for
a test environment to become
available, and further time
trying to undo all the changes
made be the previous team”

9. 9 Copyright 2014.
Continuous Delivery, DevOps &Tooling
▪ It’s been said many times: “Continuous Delivery & DevOps >> Tooling”
▪ In practice, the majority of CD & DevOps initiatives are automation initiatives
− Often, localized activity driven by individual teams or Tech Heroes
▪ Automation addresses many of the immediate pain points felt by teams
“Many of the problems in
our code only show up in
pre-production, because the
developers can’t put together
a local environment that’s
sufficiently production-like”
“We spend ages waiting for
a test environment to become
available, and further time
trying to undo all the changes
made be the previous team”
“Changes committed by
developers keep conflicting with
work made by other team
members, breaking our code”

10. 10 Copyright 2014.
Continuous Delivery, DevOps &Tooling
▪ It’s been said many times: “Continuous Delivery & DevOps >> Tooling”
▪ In practice, the majority of CD & DevOps initiatives are automation initiatives
− Often, localized activity driven by individual teams or Tech Heroes
▪ Automation addresses many of the immediate pain points felt by teams
“Many of the problems in
our code only show up in
pre-production, because the
developers can’t put together
a local environment that’s
sufficiently production-like”
“We spend ages waiting for
a test environment to become
available, and further time
trying to undo all the changes
made be the previous team”
“Changes committed by
developers keep conflicting with
work made by other team
members, breaking our code”
“Our test results are largely
useless because the test data
is not representative of
production”

11. 11 Copyright 2014.
Continuous Delivery, DevOps &Tooling
▪ It’s been said many times: “Continuous Delivery & DevOps >> Tooling”
▪ In practice, the majority of CD & DevOps initiatives are automation initiatives
− Often, localized activity driven by individual teams or Tech Heroes
▪ Automation addresses many of the immediate pain points felt by teams
“Many of the problems in
our code only show up in
pre-production, because the
developers can’t put together
a local environment that’s
sufficiently production-like”
“We spend ages waiting for
a test environment to become
available, and further time
trying to undo all the changes
made be the previous team”
“Changes committed by
developers keep conflicting with
work made by other team
members, breaking our code”
“Our test results are largely
useless because the test data
is not representative of
production”
“There are all these cool tools out there
that I want to play with”

12. 12 Copyright 2014.
Continuous Delivery, DevOps &Tooling
▪ It’s been said many times: “Continuous Delivery & DevOps >> Tooling”
▪ In practice, the majority of CD & DevOps initiatives are automation initiatives
− Often, localized activity driven by individual teams or Tech Heroes
▪ Automation addresses many of the immediate pain points felt by teams
“Many of the problems in
our code only show up in
pre-production, because the
developers can’t put together
a local environment that’s
sufficiently production-like”
“We spend ages waiting for
a test environment to become
available, and further time
trying to undo all the changes
made be the previous team”
“Changes committed by
developers keep conflicting with
work made by other team
members, breaking our code”
“Our test results are largely
useless because the test data
is not representative of
production”
“There are all these cool tools out
there that I want to play with”

13. 13 Copyright 2014.
The Scope of an Enterprise CDToolscape
▪ “Making DevOps compatible with the rest of the organization”
Your Enterprise CD Toolscape needs to go beyond the tech teams

14. 14 Copyright 2014.
The Scope of an Enterprise CDToolscape
▪ “Making DevOps compatible with the rest of the organization”
▪ Team-level DevOps & CD initiatives can become “black holes”:
great stuff going on inside, no information coming out
Your Enterprise CD Toolscape needs to go beyond the tech teams

15. 15 Copyright 2014.
The Scope of an Enterprise CDToolscape
▪ “Making DevOps compatible with the rest of the organization”
▪ Team-level DevOps & CD initiatives can become “black holes”:
great stuff going on inside, no information coming out
▪ As DevOps adoption scales, the tooling and process starts to
touch “traditional” IT & business processes
− Portfolio management
− Service management
− …
Your Enterprise CD Toolscape needs to go beyond the tech teams

16. 16 Copyright 2014.
The Scope of an Enterprise CDToolscape
▪ “Making DevOps compatible with the rest of the organization”
▪ Team-level DevOps & CD initiatives can become “black holes”:
great stuff going on inside, no information coming out
▪ As DevOps adoption scales, the tooling and process starts to
touch “traditional” IT & business processes
− Portfolio management
− Service management
− …
▪ Moving from “dev to prod” to “concept to cash”: need the ability to “track and
trace” business-relevant IDs through the toolchain
Your Enterprise CD Toolscape needs to go beyond the tech teams

17. 17 Copyright 2014.
The Scope of an Enterprise CDToolscape
▪ “Making DevOps compatible with the rest of the organization”
▪ Team-level DevOps & CD initiatives can become “black holes”:
great stuff going on inside, no information coming out
▪ As DevOps adoption scales, the tooling and process starts to
touch “traditional” IT & business processes
− Portfolio management
− Service management
− …
▪ Moving from “dev to prod” to “concept to cash”: need the ability to “track and
trace” business-relevant IDs through the toolchain
▪ Link features to user-level monitoring
Your Enterprise CD Toolscape needs to go beyond the tech teams

18. 18 Copyright 2014.
Understanding Continuous DeliveryTools
This is a very confusing and fast-moving space:

19. 19 Copyright 2014.
Understanding Continuous DeliveryTools
You most likely already have pretty much every possible tool running
somewhere
Every team will be pushing to either keep their solution
Or to make their solution the “standard”
“Let 100 CD toolsets bloom”?

20. 20 Copyright 2014.
Understanding Continuous DeliveryTools
▪ Becomes infeasible as DevOps & CD adoption
scales up
▪ Lack of expertise:
“not enough Tech Heroes to go around”
▪ Hidden maintenance overhead
• “We need to spend some time in this sprint to fix our
CI server”
▪ Difficult to implement concerns that cut across
teams
• Security
• Audit/compliance
• Integrations
• Data access
• …

21. 21 Copyright 2014.
▪ Choose one or two standard tools in each CD tool category
− Consider also cloud-hosted services if these meet security etc. requirements
▪ Create a dedicated team to provide these tools “as a service” to teams
− Usually part of the Operations/platform team, although often initially staffed with Tech Heroes from
dev/release engineering
▪ Additional responsibility: provide guidance and onboarding support to teams not
familiar with CD tools
▪ In the initial standardization phase, provide migration assistance to teams that
are happy to move to a standard tool
Recommendation: “Shared services, with exceptions”
Understanding Continuous DeliveryTools

22. 22 Copyright 2014.
Product Evaluation Criteria for Enterprise Use
Classes
“Process” “Component”

23. 23 Copyright 2014.
Product Evaluation Criteria for Enterprise Use
“Process” “Component”
Categories
Issue tracking
Continuous Integration
Release coordination/Continuous Delivery
Management
User-level monitoring
Team collaboration
SCM
Artifact repository
Cloud management
Environment provisioning
Application release automation
Test data & test result management
System-level monitoring

24. 24 Copyright 2014.
Product Evaluation Criteria for Enterprise Use
• Continuous Integration
• Cloud management
• System-level
monitoring
Commodity
• Issue tracking
• SCM
• Artifact repository
• Environment
provisioning
Established
• Application release
automation
• Test data & test result
management
• Release
coordination/CDM
• User-level monitoring
• Team collaboration
Growth
Maturity

25. 25 Copyright 2014.
1
Broad applicability of your standard tools you choose: scaling out DevOps &
CD means being able to support your mainframe wrapper as well as your “new
build” apps
• “DevOps in the Dark Corners”
Product Evaluation Criteria for Enterprise Use
2
Especially for Growth tools, favor products that have been around for a while,
unless there are exceptionally clear business reasons.
• Lots of bleeding edge tools in this area will not survive, and will require extensive
hand-holding
• Very new tools are what the teams should be researching and, if desired,
requesting exceptions for
3
For Established or Commodity tools: no need for lengthy bake-offs. Choose the 1 or 2
most popular tools in use by your teams; validate enterprise scalability through expert
sources and/or testing

26. 26 Copyright 2014.
4
For Component tools, look for APIs, strong integrations and an open extension
model
• Component tools will become increasingly “invisible” but need to be tightly
integrated with each other
5
For Process tools, look for user-friendly interfaces that are not just “suitable for
Tech Heroes”, and visualization capabilities
• The closer to the process level, the more non-technical users the tool will have as DevOps
scales out
6 For Process tools, look for integrations with “adjacent” IT and business processes:
portfolio management, service management, change management
Product Evaluation Criteria for Enterprise Use
7 All tools need to be automatically installable and configurable
• Based on a versionable configuration

27. 27 Copyright 2014.
8
All tools need to support the ability to version any created artifact
• Whether binary deliverable or process definition
9
All tools need to support the ability to “partition” the tool’s domain model according to
teams, departments etc.
• “Multi-tenant lite”
10 All tools need to support the ability to apply security settings to these partitions
Product Evaluation Criteria for Enterprise Use
11
All tools need to provide published reporting APIs or other supported means of data
access
• No matter how good the reporting capability of the tools, you will need to get data out to
answer questions relating to multiple tools, e.g. providing audit information

28. 28 Copyright 2014.
3 Real-world Examples
TFS
Jenkins
XL Deploy
Azure (PaaS)
VMware
MSBuild
Lync
Selenium
Scripts
SVN

29. 29 Copyright 2014.
3 Real-world Examples
Key Points
▪ Multiple types of target environments: PaaS and “in-house platform”
▪ One end-to-end orchestration tool
▪ One main test tool so no need for test result aggregation

30. 30 Copyright 2014.
3 Real-world Examples
VersionOne
Jenkins
Ansible
AWS EC2
Gradle
Slack
In-house DBGit
XL Release
Nexus
S3
Puppet

31. 31 Copyright 2014.
3 Real-world Examples
Key Points
▪ Two orchestrators for the “technical” and “process-heavy” parts of the software
delivery process
− Also happening at different frequencies
▪ In-house developed test database
▪ Migrating from Puppet to Ansible
▪ Considering moving away from a “traditional” artifact repo (Nexus -> S3?)

32. 32 Copyright 2014.
3 Real-world Examples
JIRA
Jenkins
Docker
OpenStack
Maven
Confluence
Email
XL TestViewSVN
ServiceNow
Docker Registry

33. 33 Copyright 2014.
3 Real-world Examples
Key Points
▪ Container-based approach but still running on a “traditional” cloud management
platform
▪ No integrated team collaboration tool
▪ Investigating container orchestration frameworks to handle challenges in
tracking container dependencies in the pipeline

34. 34 Copyright 2014.
Next Steps

35. 35 Copyright 2014.
Next Steps
▪ Download the guide:
go.xebialabs.com/IT-Managers-Guide-to-CD.html
▪ Learn more about XebiaLabs & Jenkins:
xebialabs.com/solutions/jenkins/
▪ Stay informed:
blog.xebialabs.com
@XebiaLabs
youtube.com/xebialabs

36. Questions?

37. Thank You!