The document discusses strategies to avoid deployment failures and improve recovery processes in IT environments, emphasizing the importance of planning for recovery and employing deployment models such as canary, blue/green, and rolling deployments. It highlights the significance of continuous delivery, automation, and collaboration in accelerating software releases while maintaining quality. Furthermore, it outlines best practices for managing deployments and measuring DevOps progress to enhance overall delivery efficiency.

1. 1IBM
_
Chapter
Opening
September 16, 2015Presentation Title
Shift Happens
Continually moving forward when the
outcome looks bleak
@Al_Wagner

2. 2IBM
_
Avoiding
Deployment
Failures..
especially those that could cause a production outage, is top of
mind for many IT professionals. However, sometimes failures will
occur in production, which means that planning for recovery is
essential. Preventative measures like canary, blue/green or rolling
deployments can help, but also having the ability to roll forward
(instead of rolling back), also known as shifting right, means you can
push through a failure while learning from deployment process
mistakes and shortening mean time to recovery (MTTR).
• Deployment models like canary, blue/green and rolling that can
help prevent major production outages
• How to pinpoint deployment failures in your process and correct
them
• Pulling together a basic failure response plan
• How you can roll forward while improving your deployment
process
September 16, 2015Shift Happens

3. Survey says…
https://www-01.ibm.com/marketing/iwm/dre/signup?source=mrs-form-3570&S_PKG=ov50501

4. DevOps is all about executing with speed!
Line-of-
business
Customer
Getting ideas into production fast – Getting people to use it – Analyzing their feedback
Continuous Delivery
Continuous Feedback
Continuous Innovation

5. • Reducing Scope
ü Small batches of
incremental changes
• Empowering Resources
ü Co-located,
automatous teams
• Accelerating Schedules
ü Automate, automate,
automate
• Increasing Quality
ü Everyone contributes
ü Small batches of
incremental changes
ü Co-located, automatous
teams, collaboration
ü Continuous release &
deployment
ü Everyone contributes
5
Managing the Iron Triangle by…
Quality
Schedule
Scope
Resources

6. Traditional software deployments
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Users
Environment #1
Environment #2
Load
Balancer
1. Servers taken off-line

7. Traditional software deployments
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Users
Environment #1
Environment #2
Load
Balancer
New deployment is tested
1. Servers taken off-line
2. New release is deployed & tested
The clock is ticking!
Software
Deployment

8. Traditional software deployments
Web
Server
App
Server
Database
Server
App
Server
Database
Server
Users
Environment #1
Environment #2
Load
Balancer
New version of application
1. Servers taken off-line
2. New release is deployed & tested
3. Servers brought back on-line
Web
Server

9. Manual deployments are error prone
9
One wrong move and
it can all

10. And when disaster strikes! You need to know…
What
failed?
Where did
it fail?
What apps
were
impacted?
Should I move
traffic to
another server?
Do we go
forward or
rollback?
If you fail to plan;
you plan to fail!
Why did
it fail?

11. During the post mortem, you need to uncover…
Did anything trigger the deployment failure?
What was the root cause of the failure?
What could we have done differently to avoid this situation?
How can we improve so it doesn’t happen again?

12. Accelerate delivery of incremental software change
Failures due to
inconsistent dev
and production
environments
Bottlenecks trying
to deliver more
frequent releases
to meet market
demands
Complex, manual,
processes for
release lack
repeatability and
speed
Poor visibility into
dependencies
across releases,
resources, and
teams

13. Accelerate delivery of incremental software change
Failures due to
inconsistent dev
and production
environments
Bottlenecks trying
to deliver more
frequent releases
to meet market
demands
Complex, manual,
processes for
release lack
repeatability and
speed
Poor visibility into
dependencies
across releases,
resources, and
teams
The Four Pillars of
Gold-Standard Deployment
• Use the same process
ü Reduces deployment errors
• Automate, automate, automate
ü Deliver repeatability, reliability, &
with traceability
• Deliver incremental changes
ü Reduces risk to business
• Release what you test
ü Increases confidence

14. Automate provisioning and deployments
SCM
Build
Automation
Publish build
Pull
changes
IBM Cloud Orchestrator
IBM PureApplication System
IBM Cloud Manager
with OpenStack
IBM Bluemix
Provision environment
with open patterns
Public: Shared
off premises cloud
Dedicated:
off premises cloud
Local: Dedicated
on premises
cloud
Traditional IT
ü Traceable
VMWare
vCenter
ü Repeatableü Reliable
IBM UrbanCode
Deploy
Automate
deployment to
hybrid environments

15. IBM Cloud UrbanCode Deploy as a Service
Develop Build
Mobile Device
Mainframe
Traditional
Deploy
Features of the new SaaS offering
• Full automated application delivery capabilities
• Hosted on IBM infrastructure, managed by IBM
• Monthly subscription, license managed by IBM
• Full product support
App
App
App
App
SoftLayer, AWS, Azure
App
IBM Cloud
UrbanCode
Deploy
NEW!

16. 16Page© 2016 IBM Corporation
IBM UrbanCode Release for release management
1
ü No more release week-end
parties: Coordinate
stakeholders, orchestrate
deployment activities, enforce
qualification process with
relevant workflow and quality
gates, get necessary approvals
prior getting to production.
Make releases predictable and
boring!
ü Reduced down time: Eliminate
wasted time, orchestrate large
& complex releases involving
several hundred applications,
and hundreds of stakeholders.
ü Reduced time to market with
continuous delivery releases:
Accelerate release frequency
with distributed release
management for small scope
frequent releases delivered by
application teams
Make releases predictable and boring!

17. IBM UrbanCode Release & Deploy iOS mobile app
ü Monitor Progress:
Understand the overall
progress of your releases
and remaining work. Get
real time calculations of
the projected completion
time
ü Alert for Critical
issues: See critical data
of late tasks and idling
tasks so you can
encounter problems and
mitigate business risks.
ü Understand team
status: Learn from teams
what they are blocked by
to take the right corrective
actions
https://itunes.apple.com/ca/app/ibm-urbancode-release-deploy/id1084753666?mt=8

18. Shift right and continuously move forward
Accelerate releases by making a conscious
decision to carry an acceptable level of …
…into PRODUCTION!

19. Dark Launches & Toggles
• Feature toggle - restricts access to source code
in development until ready for release to end
users
if “work_in_progress” {
develop new functionality here
} else {
already deployed as production code
};
• Business toggle – control user or group of user
access to new functionality
if “beta_usergroup” {
provide access to new experiment
} else {
route user to existing production code
};
ü Pros
New experiments can
quickly be made
available to groups of
trusted users
X Cons
Increase in technical
debt as ”toggle” code
needs to be managed

20. Zero downtime deployment strategies
Canary Release Blue/Green Deployments Rolling Deployments
a technique to reduce
the risk of introducing a
new software version in
production by slowly
rolling out the change
to a small subset of
users before rolling it
out to the entire
infrastructure and
making it available to
everybody.
a release technique
that reduces downtime
and risk by running two
identical production
environments
called Blue and Green.
At any time, only one of
the environments is
live, with the live
environment serving all
production traffic.
a software release
strategy that staggers
deployment across
multiple phases, which
usually include one or
more servers
performing one or more
functions within
a server cluster to
reduce application
downtime.

21. Canary Releases (example flow)
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Users
Old Version
Old Version
50% of
Users
Load
Balancer
50% of
Users

22. Canary Releases (example flow)
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Users
Old Version
New Version
All
Users
Deployment
AutomationInventory
Load
Balancer

23. Canary Releases (example flow)
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Users
Old Version
New Version
Most
Users
(95%)
Some
Users
(5%)
Deployment
AutomationInventory
Load
Balancer
As confidence in the new release
increases, the percentage of users
who have access is increased.

24. Canary Releases (example flow)
Web
Server
App
Server
Database
Server
Users Load
Balancer
Old Version
Web
Server
App
Server
Database
Server
New Version
All
Users
Deployment
AutomationInventory
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
New VersionNew Version
Eventually the new version is
deployed to the second
environment.

25. Canary Releases (example flow)
Web
Server
App
Server
Database
Server
Users Load
Balancer
Old Version
Web
Server
App
Server
Database
Server
New Version
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
New VersionNew Version
50% of
Users
50% of
Users
And the user load is split across the
two environments.

26. Blue / Green Deployments (example flow)
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Environment #1
RouterUsers
All
Users
Two environments, each of
sufficient resources to serve the
application in production.
Environment #2
Previous Release
(hot stand-by)

27. Blue / Green Deployments (example flow)
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Environment #1
Environment #2
RouterUsers
All
Users
Two environments, each of sufficient resources
to serve the application in production.
Deployment
AutomationInventory
The new release is
deployed to the idle
environment.

28. Blue / Green Deployments (example flow)
Web
Server
App
Server
Database
Server
Web
Server
App
Server
Database
Server
Environment #1
RouterUsers
All
Users
Two environments, each of sufficient resources
to serve the application in production.
Environment #2
Previous Release
(hot stand-by)
When the new deployment is
working as expected, users are
routed to the new version.

29. Load
Balancer
Rolling Deployments (example flow)
Web
Server
App
Server
Database
Server
Server Cluster #1
Web
Server
App
Server
Database
Server
Server Cluster #2
Web
Server
App
Server
Database
Server
Server Cluster #3
Web
Server
App
Server
Database
Server
Server Cluster #4
Users

30. Rolling Deployments (example flow)
Web
Server
App
Server
Database
Server
Server Cluster #1
Web
Server
App
Server
Database
Server
Server Cluster #2
Web
Server
App
Server
Database
Server
Server Cluster #3
Web
Server
App
Server
Database
Server
Server Cluster #4
Users
Deployment
AutomationInventory
Load
Balancer
1. Cluster #1 taken off-line
2. Application change deployed
3. Deployment tested

31. Rolling Deployments (example flow)
Web
Server
App
Server
Database
Server
Server Cluster #1
Web
Server
App
Server
Database
Server
Server Cluster #2
Web
Server
App
Server
Database
Server
Server Cluster #3
Web
Server
App
Server
Database
Server
Server Cluster #4
Users
Deployment
AutomationInventory
Load
Balancer
1. Cluster #1 brought back on-line
2. Cluster #2 is taken off-line
3. Application change deployed
4. Deployment tested

32. Rolling Deployments (example flow)
Web
Server
App
Server
Database
Server
Server Cluster #1
Web
Server
App
Server
Database
Server
Server Cluster #2
Web
Server
App
Server
Database
Server
Server Cluster #3
Web
Server
App
Server
Database
Server
Server Cluster #4
Users
Deployment
AutomationInventory
Load
Balancer
1. Cluster #3 brought back on-line
2. Cluster #3 & #4 is taken off-line
3. Application change deployed
4. Deployment tested

33. Rolling Deployments (example flow)
Web
Server
App
Server
Database
Server
Server Cluster #1
Web
Server
App
Server
Database
Server
Server Cluster #2
Web
Server
App
Server
Database
Server
Server Cluster #3
Web
Server
App
Server
Database
Server
Server Cluster #4
Users
Load
Balancer
All environments are presenting
the latest version of the
application.

34. Pros and Cons…
Canary Release Blue/Green Deployments Rolling Deployments
Pros
• No downtime of production
environment
• Quick access to a backup
environment
• A/B testing of new features and
functionality
• Capture performance metrics of
new release during early adoption
Cons
• Management and maintenance of
multiple versions of the software
• Maintain persistent sessions
during deployment
• Database must support two
versions of the application (until
cut-over is complete)
Pros
• No downtime of production
environment
• Quick access to a backup
environment – hot standby
• Ability to test application in a
production environment
Cons
• Requires two similar environments
• Maintain persistent sessions
during deployment
• Database must support two
versions of the application (until
cut-over is complete)
Pros
• No downtime of production
environment
• Incrementally validate
deployments and reduce risk
• Reduce visibility of performance
degradation
• Seamless user experience
Cons
• Maintain persistent sessions
during deployment
• Database must support two
versions of the application (until
deployment is complete)

35. Your mission if you choose to accept it…
Measure your DevOps progress
• Deployment / Change Frequency
– Measures delivery team responsiveness, cohesiveness, capabilities, efficiency, & tooling
effectiveness
• Change Lead Time
– Measure efficiency of end to end development process; from first code change to deployment
– Measure cycle time of the individual activities
• Change Failure Rate
– How many deployment fail / number of deployments
• Mean Time To Recover (MTTR)
– How long does it take to recover from a failure
– Understand the contributors to failure:
• code complexity, number of app changes, number of operating environment changes

36. 3

37. 37IBM
_
September 16, 2015Shift Happens
Thank You