2. Please note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in
a controlled environment. The actual throughput or performance that any user will experience
will vary depending upon many factors, including considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration,
and the workload processed. Therefore, no assurance can be given that an individual user
will achieve results similar to those stated here.
4. 4
CEOs identify technology as the most important
external force impacting their organizations – again
Source: IBM Institute for Business Value, The Global CEO Study 2013. Question: “What are the most important external
forces that will impact your organization over the next 3 to 5 years?”
2004 2006 2008 2010 2012 2013
1
2
3
6
4
5
7
8
9
1
2
3
6
4
5
7
8
9
1
2
3
6
4
5
7
8
9
1
2
3
6
4
5
7
8
9
1
2
3
6
4
5
7
8
9
Technology Factors
Market Factors
Macro-economic Factors
People Skills
Regulatory Concerns
Socio-economic Factors
Globalization
Environmental Issues
Geopolitical Factors
2
3
6
4
5
7
8
9
1
CEO Studies 2004–2013
External forces that will
impact the organization
5. Software
delivery Intelligent/
Connected Systems
Software component in
smart products driving
increased value and
differentiation
Big Data
Insights on new products by
more efficiently interpreting
massive quantities of data
Cloud
Demand for apps requires fast,
scalable environments for dev
and test, as well as production
Instrumented Products
Industry requirements demand
faster response to regulations
and standards, with traceability
and quality
Social Business
Broader set of stakeholders
collaborates to deliver
continuous innovation
and value
Mobile
Modern workforce
expects constantly
updated software to
connect to enterprise
systems
Software delivery is at the heart of today’s top
technology trends
11. A lack of continuous delivery impacts the entire
business enterprise in the new reality of “Systems Of
Interaction”
>45%
of customers
experience
production delays
>50%
of outsourced
projects fail to
meet objectives
>70%
of budgets devoted
to maintenance
and operations
4-6
weeks
to deliver even minor
application changes
to customers
Systems of Interaction
Continuous
client
experience
Partner
value chain
Cloud-based
Services
Systems of
Engagement Systems of Record
Operations
Rapid app releases impact
system stability and compliance
Suppliers
Delivery in the context of
agile
Development/Test
Speed mismatch between faster moving
front office and slower moving back office
systems, delaying time to obtain feedback
Line-of-business
Takes too long to introduce or make
changes to mobile apps and services
HR
DB ERP
MF iSeries
CRM
12. Agenda
• Why DevOps?
• What is DevOps?
• Adopting DevOps
o People
o Process
o Technology
13. DevOps is a Philosophy
Source: http://virtualplatosacademy.blogspot.com/p/great-philosophers.html
14. Getting Started - The Critical Measure of DevOps Success
The Hidden Factory Opportunity
Overhead
Productive
80%
20%
50%
50%
Hidden Factory= additional value you could create if you eliminated waste and redirected those resources at innovation
DevOps
Transformation
15. The Big Sources of Wasted Efforts: Find the Hidden
Factory
Forrester Research study finds an average 3x project capacity after implementing
DevOps
Type of Waste Create Feature Deliver Feature
Unnecessary
Overhead
Communicating ideas/knowledge
Supporting Artifacts
Communicating between development
and operations
Unnecessary Re-
work
Tasks assigned back to developers
from testing and usage
Tasks assigned back to developers from
production rollbacks
Over-production Unnecessary functionality produced Unnecessary hardware, data center,
personnel
Forrester Consulting: Total Economic Impact of IBM DevOps
The study analyzed several development teams and found that the
group, on average, went from delivering 10 projects in Year 1 before
implementing IBM DevOps, to 20 projects in Year 2, to 30 projects in Year
3 — all without having to add additional developer headcount.
And the developers were happier, less stressed, and had a greater
feeling of accomplishment.
16. DevOps approach: Apply Lean principles accelerate
feedback and improve time to value
Line-of-
business
Customer
1
3
2
1. Get ideas into production fast
2. Get people to use it
3. Get feedback
Adopt DevOps approach to
continuously manage changes,
obtain feedback and , deliver
changes to users
Change speed must be an asset,
Not an anchor
Non-Value-added waste
Value-added production work
Lean
Transformation
17. Business Outcomes and Operational Objectives
§ Faster Time to Value
– Reduced cycle times
– Industrialization
§ Improved predictability
– Operational excellence
– Uncertainty reduction
§ Improved customer satisfaction
– Customer alignment
– Transparency in real time
§ Improved efficiency
– Waste reduction, reduced overhead, rework, overproduction
– Capex / Opex optimization
18. IBM DevOps Adoption Model
Practices, tools and services to plan and execute a staged adoption of DevOps to
improve business outcomes
Feedback
Cycles
Productive Waste
Efficiency
Steer Product-based
Agile
Automated
Collaborative
Optimizing
More
Predictable
More
Transparent
More
Continuous
Process-based
Process-heavy
Manual
Silo-ed
Develop/Test
Deploy
Operate
Inefficient Leaner Leaner and Smarter
19. Map your Bottlenecks
Idea/Feature/Bug Fix/
Enhancement
Production
Development Build QA SIT UAT Prod
PMO
Requirements/
Analyst
Developer
CustomersLine of Business
Build
Engineer
QA Team Integration Tester User/Tester Operations
Artifact Repository
Deployment Engineer
Release Management
Code Repository
Deploy
Get Feedback
Infrastructure as Code/
Cloud Patterns
Feedback
Customer or
Customer Surrogate
Metrics - Reporting/Dashboarding
Tasks
Artifacts
20. Key software delivery bottlenecks we must eliminate
20
Lack of alignment between IT and business goals
Complexity of languages, tools, and platforms
Poorly integrated teams across the application lifecycle
Lack of automated and scalable testing methods
High cost of setting up and maintaining test environments
Application release process is still error-prone and slow
Lack of centralized incident management and monitoring
Slow or unresponsive to customer feedback
Deploy
Develop
- Test
Steer
Operate
Bottlenecks impact delivery cycles, cause rework, and waste resources
21. The IBM DevOps Adoption Model
IT Org leader Team member
One-on-one guidance on how to improve, from where
you are.
How do I
determine
the best
place to
start?
I know where I
want to start.
What’s the
fastest way to
success?
Inefficient Leaner Leaner and Smarter
CollaborativeSilo-ed More
Continuous
Process-
based
Process-
based Agile More
Predictable
Manual Automated More
Transparent
Steer
Develop/
Test
Operate
Deploy
OptimizingOutcome-
based
…
…
…
…
Lean Self-Assessment: http://ibm.biz/devops-
lean-assessment
DevOps Fitness Desk in Solution Center EXPO
Practice Self-Assessment: http://ibm.biz/
devops-practices-assessment
Consulting Cafe
22. Agenda
• Why DevOps?
• What is DevOps?
• Adopting DevOps
o People
o Process
o Technology
24. 24
Building a DevOps Culture
• Everyone is responsible for Delivery
• Common measures of Success
• Right People are needed
Product
Owner
Team
Member
Team Lead
Team
Member
Team
Member
Senior
Executives
Users
Domain
Experts
Auditors
Gold Owner
Support Staff
External
System Team
Operations
Staff
It’s all about the People/Culture
26. Agenda
• Why DevOps?
• What is DevOps?
• Adopting DevOps
o People
o Process
o Technology
27. Adopting DevOps in the Enterprise: Process
27
• DevOps as a Business Process
• A Process to get Capabilities from Ideation to Value
• Apply Lean Thinking to Processes
28. Key Capabilities
1. Collaborative Development & Continuous
Integration
2. Continuous Business Planning
3. Continuous Release and Deploy
4. Continuous Testing
5. Continuous Feedback
Adopting DevOps in the Enterprise: Process
29. 1. Collaborative Development and Continuous Integration
http://bit.ly/PRQ4a7
Mobile App
Developent
Teams
Enterprise
Services
Developent
Teams
Adopting DevOps in the Enterprise: Process
30. 2. Continuous Business Planning
3. Continuous Release and Deploy
4. Continuous Testing
5. Continuous Feedback http://bit.ly/PRQ4a7
Adopting DevOps in the Enterprise: Process
31. Agenda
• Why DevOps?
• What is DevOps?
• Adopting DevOps
o People
o Process
o Technology
32. • Infrastructure as Code/Software
Defined Environments
package "apache2" do
package_name node['apache']['package']
end
service "apache2" do
case node['platform_family']
when "rhel", "fedora", "suse"
service_name "httpd"
# If restarted/reloaded too quickly httpd has a
habit of failing.
# This may happen with multiple recipes
notifying apache to restart - like
# during the initial bootstrap.
restart_command "/sbin/service httpd restart &&
sleep 1"
reload_command "/sbin/service httpd reload &&
sleep 1"
/* REXX */
/* REXX BIND processor sample */
trace o
Arg PACKAGE DBRM
rcode = 0
/* Set BIND options */
SYSTEM = 'DSN9'
i = Pos('(', DBRM)
len = Length(DBRM)
LIBRARY = Substr(DBRM, 1, i - 1)
MEMBER = Substr(DBRM, i + 1, len - i - 1)
OWNER = 'DEVDBA'
ACTION = 'REPLACE'
VALIDATE = 'RUN'
ISOLATION = 'CS'
EXPLAIN = 'NO'
QUALIFIER = 'DEVDBA'
Call Bind_it
Exit rcode
Bind_it:
/* Create a bind control statement as a single long line. Then */
/* queue that into a FIFO stack */
DB2_Line = "BIND PACKAGE("PACKAGE")" ||,
" LIBRARY('"LIBRARY"')" ||,
" MEMBER("MEMBER")" ||,
" OWNER("OWNER")" ||,
" ACTION("ACTION")" ||,
" VALIDATE("VALIDATE")" ||,
" ISOLATION("ISOLATION")" ||,
" EXPLAIN("EXPLAIN")" ||,
" QUALIFIER("QUALIFIER")"
/* Write the bind control statement to the data queue and execute */
/* DB2I to perform the bind. */
queue DB2_Line
queue "End"
Address TSO "DSN SYSTEM("SYSTEM")"
rcode = RC
Return
Adopting DevOps in the Enterprise: Technology
33. Adopting DevOps in the Enterprise: Technology
33
• Common Collaboration Tools
• Common Work Item Management Tool
• Dashboards to show status/progress
34. Develop /
Test DeploySteer Operate
IBM DevOps – Broad set of DevOps capabilities
Address bottlenecks across the application delivery lifecycle
Ra#onal
Focal
Point
Ra#onal
DOORS
NG
Ra#onal
Team
Concert
Ra#onal
Lifecycle
Integra#on
Adapters
Ra#onal
Quality
Manager
Ra#onal
Test
Workbench
Ra#onal
Test
Virtualiza#on
Server
InfoSphere
Op#m
Test
Data
Management
IBM
UrbanCode
Release
IBM
UrbanCode
Deploy
SmartCloud
Orchestrator
SmartCloud
Applica#on
Performance
Management
SmartCloud
Control
Desk
Continuous
Business Planning
Collaborative
Development
Continuous
Testing
Continuous Release
and Deployment
Continuous
Monitoring
Continuous
Customer Feedback
& Optimization
IBM
Tealeaf
IBM
Digital
Analy#cs
“IBM has a more comprehensive
end-to-end DevOps and ALM
testing integration tool strategy, and
the strongest market momentum.” –
Diego Lo Giudice, Forrester Research,
January 27, 2014
35. Implementing a DevOps toolchain
SCM
Build / CI
Server
Unit testing
Test Automation
Test Stubbing
Delivery
Pipeline
Environment
Configuration
Automated
Monitoring
Asset
Repository
36. UrbanCode for Release Automation
IBM UrbanCode Deploy automates the deployment of applications,
databases and configurations into development, test and production
environments, helping to drive down cost, speed time to market with
reduced risk.
IBM UrbanCode Release is an intelligent collaboration release
management solution that replaces error-prone manual spreadsheets
and streamlines release activities for application and infrastructure
changes.
https://www.ibmdw.net/urbancode/
Drive down cost
Remove manual effort and wasted resource time with
push button deployment processes
Speed time to market
Simple, graphical process designer, with built-in
actions to quickly create deployment automation
Reduce risk
Robust configuration management, coordinated
release processes, audits, and traceability
38. Databases Internal
Messages
Third-party
Services
virtual components
Simultaneously
test across
multiple test
stages
Dev QA
IBM Rational Test
Virtualization
Server
Continuous Testing: Deploy what is ready, Virtualize the rest
IBM UrbanCode
Deploy
IBM Rational
Test Workbench
{
integrated with
Test Environments
Dynamic Infrastructure
Deploy what is ready,
virtualize the rest
Continuously test in
production-like env.
Test using real world
network conditionsn
Network
Virtualization
39. Cloud Portability with Patterns
Cloud
Resources
(DEV)
Cloud
Resources
(PROD)
Environments
QA
...
PROD
World-Wide Banking Application
war
ddl
mq
World-Wide Banking Application
war
ddl
mq
IBM
Virtual
System
Pa2erns
OpenStack
HOT
templates
World-Wide Banking Application
war
ddl
mq
DEV
UrbanCode
Deploy
Cloud
ApplicaAon
Component
Tomcat
MySQL
JMS
WebSphere
Liberty
DB2
WSMB
SmartCloud
Orchestrator
IBM
PureApplicaAon
System
IBM UrbanCode Deploy
Private CloudsHybridPublic Clouds
VM VM VM
40. Using Patterns to define ‘Full Stack’
Environments
What’s required…What the business wants… What a pattern automates…
Patterns of Expertise:
Proven best practices and expertise
learned from decades of client and
partner engagements
• Pre-defined architecture of an application or
Cloud service
• Captures best practices for complex tasks
• Optimized into a deployable form for private
or public cloud
• Repeatable deployment with full lifecycle
management reducing operational costs
41. • Organic environments
• Version-aware
• Full-stack engineering
• Composable content
• Cloud agnostic
IBM UrbanCode Deploy with
Patterns
HOT document editor & Environment
lifecycle management tool
A full-stack engineering solution for designing, deploying, and managing environments
Components
Resources
Heat Templates
(HOT)
Building
Blocks
HOT
Executes on
Next generation pattern development and lifecycle
management
OpenStack Heat
Tool
Engine
Platforms
42. o Provide Software Components right
along side infrastructure
o Represent these resources in HOT
documents
o Once deployed update either from
Heat or UrbanCode Deploy
o Will support Chef, Puppet, etc over
time
IBM UrbanCode Deploy & Heat
Templates
43. Improve Delivery Lifecycle with IBM Virtual System Patterns
SCM Build Automation
IBM Endpoint
Manager
QA PROD
Provision platformExecute application
deployment and
manage settings
across environments
Patch/update
Publish build
Pull
changes
SmartCloud
Orchestrator
Platform Config Management
Environments
Deploy early and
often to ensure
higher quality and
faster releases
using repeatable,
reliable, and
managed
automation
DEVDEVDEVDEV
IBM
PureApplicaAon
System
44. 44
§ DevOps Lean Assessment (Beta):
http://ibm.biz/devops-lean-assessment
§ IBM DevOps Page: http://ibm.com/DevOps
§ DevOps For Dummies Book:
http://ibm.co/devopsfordummies
§ Release and Deploy For Dummies Book:
http://ibm.co/1bplaQV
§ IBM DevOps YouTube Playlist: http://bit.ly/1fiDOtl
Resources
46. Thank You!
Your Feedback is Important!
Access the Innovate agenda tool to complete your session
surveys from your smartphone, laptop or conference kiosk.
Editor's Notes
In terms CEO priorities, technology factors continue to dominate as the major external factor impacting their businesses.
According to CEOs we interviewed in our 2004 CEO study (left hand side of the slide), technology was not even among the top-five forces impacting their organizations.
By 2012, however, CEOs cited technology as the number-one factor they see impacting their companies … in front of people skills, market factors and macroeconomic factors.
Clearly, it is top-of-mind for today’s business leaders – and likely will remain so for the foreseeable future.
Moving on to the next slide …
Additional Notes:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Source: IBM Institute for Business Value, The Global CEO Study 2012. Question asked: “What are the most important external forces that will impact your organization over the next 3 to 5 years?”
Note: The actual percentages for the top three factors are (if asked): Technology factors – 71%, People skills – 69% and Market factors – 68%
Businesses who can establish software delivery as their core competency will be able to exploit the emerging technology trends
Unprecedented demands from cloud, mobile and social, and big data and analytics, instrumented products etc. are causing significant changes in how companies deliver software, requiring new levels of collaboration and innovation.
Business as usual in IT is not going to survive. Traditional software delivery methods evolved over the years across many IT industries now pose a bigger risk tan ever, resulting in loss of competitive advantage where business survival can be at stake.
TRANSITION: And majority are still taking the traditional approach when it comes to software delivery
Next slide….
2010: The Enterprise and Scrum
Results show that current software delivery methods need to change as they cannot satisfy stakeholder or business needs
A lack of continuous delivery impact the entire business. And when you combine the new requirements coming from Systems of engagement side which now have to integrate and access existing systems of record applications, the complexity dramatically increases.
Business Need: Experiment in the market to judge success of products and services. LOB: Cannot iterate quickly with market based on IT timelines
Business Need: Maintain system stability. Ops: Cannot keep up with the pace of change pushed into the system
Business Need: Quickly iterate on functionality. Dev: Cannot access production-like environments to validate application changes
TRANSITION: Status-quo is not sustainable, a new approach is needed.
Let’s go to the next slide….
Financially the issue is simple. Demands on software delivery resources are increasing and funding is decreasing. DevOps targets this dilemma: that both speed and efficiency must improve. This is why lean, agile, automation, and collaboration are key elements of any solution.
At least 40% of your resources are probably wasted on non-value added effort.
What if you could redirect this wasted resource to a Hidden Factory that dramatically improves your competitive differentiation? Hidden Factory is a Lean Six Sigma term. It is the additional output that would be possible if the resources you are currently directing at creating waste were released and redirected instead at creating value and innovation. Think about what cutting that waste by 50% or more would mean. This is the value of lean adoption and executing the core value proposition of DevOps.
Reducing waste, duplication and process friction means we can spend less time on drudgery, duplication and rework, and more time on efficient innovation and smarter systems, products and services.
IBM commissioned Forrester Consulting to interview IBM Rational clients and examine the return on investment (ROI) they achieved with CLM, Continuous testing and Service virtualization . The reports were pulibshed in July 2013
Over the years, these supporting artifacts have grown in complexity and resource burden far more than needed. Agile development methods evolved specifically to attack growth in this non-value added work. DevOps principles expand that same agile, lean thinking across the delivery lifecycle targeting the unnecessary overhead, rework and overproduction. Lean transformation is a central theme of DevOps to address this Hidden Factory opportunity.
Some overhead is a necessity, but excessive overhead, excessive rework, overproduction of unneeded function points and time wasted in waiting are hidden factory opportunities. And everyone—executives, project leadership, and practitioners KNOW they spend a huge chunk of time in non-value added work that they want to streamline, automate or eliminate. Everyone buys in to this pain, and pretty easily.
So what’s in the way?….
Nationwide is just one of many of clients who have recognized the need for change and have overcome these challenges that perhaps you may also recognize in your own organization. These are clearly not new challenges but the emergence of the 4 key technological forces (mobile, cloud, social and big data) have but additional burden on your IT organization with limited budget, resources and flexibility to quickly respond to the new business reality. Let’s take few minutes to understand how these challenges are driving up costs and increasing business risk.
Plan & Measure:
Lack of alignment between IT and business goals leading to chaotic behaviors putting entire business at risk.
Develop and Test:
Complexity of languages, tools, and platforms resulting in skills lock-in and putting additional burden on limited skilled staff and experts with deep technical experience
Poorly integrated teams across the application lifecycle creating inefficiencies, misunderstandings, source of errors, frustration, and waste
Lack of automated and scalable testing methods resulting in late discovery of breakages, creating unstable or unavailable systems, frustrating your end-users
High cost of setting up and maintaining test environments allocated to purchase, set up, configuration and maintenance of server and storage environments.
Release & Deploy:
Application release process is still error-prone and slow – and how can we forget those weekend deployment parties offering free pizza and coffee to lure us in.
Monitor & Optimize:
There is lack of centralized incident management and monitoring increasing threat and exposing business to network attack and data breaches
Slow or unresponsive to customer feedback resulting getting in unhappy customers who now can let their frustrations known to the world!
So let’s now understand how we might start to tackle some of these challenges…..
Accelerate your DevOps journey.
Are you leading an organization interested in becoming more efficient/leaner:
Take the DevOps Lean Self-Assessment to understand where your bottlenecks are: http://www.surveygizmo.com/s3/1635035/IBM-DevOps-Lean-Assessment
Come to the AgileZone in the Solution Center Expo and discuss your results with a DevOps expert who can provide recommendations on where you might focus your improvement initiatives.
Solution Center Expo Location: Dolphin Hotel - Lower Level, Atlantic Hall
Are you part of a team interested in improving your software development and delivery capabilities:
Take the DevOps Practices Self-Assessment to understand your team’s practice proficiency: http://www.surveygizmo.com/s3/1659087/IBM-DevOps-Self-Assessment
Come to the Consulting Café and discuss your results with a DevOps expert who can provide recommendations on specific DevOps practices and solutions that you may want to consider adopting.
IBM provides the most comprehensive set of DevOps capabilities -- this allows us to help our clients address bottlenecks across the application development and delivery lifecycle, with particular focus on development, testing, and deployment.
IBM leadership:
#1 vendor in Gartner market share for Application Development software for 13 consecutive years, and #1 in Evans Data Corporation User's Choice Survey for Software Development Platforms for 6th time in 7 years.
Tier 1 analysts like Gartner and Forrester recognize the breadth and category leadership of the IBM portfolio. <reference Diego Lo Guidice quote from Forrester Research>
IBM is a leader in Gartner Magic Quadrants and Forrester Waves across all key app dev segments, PLUS we have independent analyst-authored ROI studies supporting key segments.
DevOps – four adoption paths (or entry points), with defined practices, comprise DevOps approach and scope:
Plan & Measure – Continuous planning helps prioritize application portfolio decisions, measure results, and keep investments aligned with business strategy and customer input.
Develop & Test – Collaborative development and continuous testing support the rapid progression of a business idea or requirement into a high quality software solution by applying agile principles, facilitating collaboration among stakeholders, and striking the optimal balance between quality and time to market.
Release & Deploy – Helps client continuously release and deploy complex mobile, cloud, web-based and traditional applications. Client gain visibility across software releases and ensure accurate and consistent application stacks across the delivery lifecycle (development, test and operations) that results in reduced cost and risk and improved quality.
Monitor & Optimize – We help clients better understand and improve the customer experience through continuous monitoring and continuous feedback and optimization. We can help our clients meet and exceed service level agreements for app performance and availability. And improve their customer engagement experiences with mobile and web apps through sentiment analysis and other digital analytics.
IBM delivers DevOps capabilities on-prem, in the cloud, and hybrid.
The product names noted are largely on-prem solutions and hybrid solutions. Many of these capabilities are delivered as services in the cloud.
IBM DevOps Services optimized for BlueMix provide an essential set of developer tools to get teams up and running quickly without the hassles of software installations, configurations and maintenance.
IBM DevOps Services include: app auto-scaling, monitoring & analytics, mobile quality assurance, Git hosting, web IDE, continuous integration, agile planning & tracking, plus various third-party offerings – with lots more to follow.
UrbanCode improves software delivery enabling continuous release and deployment via application release automation, helping to drive down costs, speed time to market and reduce risk.
To complement IBM’s DevOps solutions already in place, UrbanCode was acquired to strengthen the Release & Deploy segment
Speak to changing delivery mindset:
Infrastructure Developer vs. Operator/Administrator
Need to bring a software development mindset to the operational areas
Replicate, where appropriate, standard architecture/development tools and methodologies
Use an Agile approach to delivery of routine tasks
Continuous, incremental improvements and delivery of new functionality
Automated unit and integration testing improves operational runtimes
Version management for scripts & source code
Automation routines and scripts are fundamental to Operations
Central point of truth as routines and environments change
Identify possible regressions by comparing with prior versions
Example Managed Assets:
Perl, Jython, WSADMIN, ANT scripts, Service orchestration routines (opsware, buildforge, etc),
Replace with new diagram – UC product logos consistent throughout
Close with bringing in the orchestration piece and how Deploy (deployment automation) combined with service virtualization, test automation, and dynamic infrastructure is the end goal.
Deploy what is ready, virtualize the rest
Continuously execute tests against production-like environments
Test end-to-end, including mobile, cloud-based applications, and mainframe