3. The Release Hurdle
2-3 weeks of code changes per release
Highly-orchestrated releases (6+ hours)
^ Same for rollbacks
Scheduled downtime
Long, stressful days
Slow iterations
4. pro·duc·tion [pruh-duhk-shuhn] (n)
1. This complex system of application code,
distributed services, servers, networking gear, etc.,
upon which we’re going to try to carefully apply a
complicated set of changes and hope that nothing
goes wrong. Cross your fingers… here goes.
5. Software for large-scale web sites has been
traditionally written by one group of people, then
released and operated by a different group.
These two groups have very different levels of
visibility into how the software works.
8. First, Principles
Enable rapid product innovation
Resolve scaling hurdles
Reduce mean-time-to-recovery
Grow a talented and engaged engineering team
Minimal bureaucracy and process
18. Every passing build is deployed to
production.
Continuous Deployment Continuous Delivery
✓
19. All enhancements are gated by
Config Flags. (“Branch in code”)
Continuous Deployment Continuous Delivery
✓ ?
20. Most of the builds we deploy are
“dark” changes.
CSS rules, template copy, un-referenced code (classes,
functions, templates), code paths behind disabled
flags, etc.
21. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
22. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
23. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
24. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production
25. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production
Assumptions:
Deploy process is infallible.
Staging is a perfect reflection of
Production, with respect to
hardware, configurations, data,
overall load, capacity, etc.
26. “What do you mean, ‘it’s not working in
production?’ I TESTED IT BEFORE WE
RELEASED!”
(DUN DUN DUUUUHHHNNN !!!)
27. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
28. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
30. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
"Because you’re integrating
so frequently, there is
significantly less back-
tracking to discover where
things went wrong , so you
can spend more time
building features.”
!
—ThoughtWorks
!
!
http://www.thoughtworks.com/continuous-integration
Dev / Integration Staging Production
31. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
Where’s the bug?
!
In one of the numerous check-ins?
Missing unit tests?
Missing automated UA tests?
Missing manual UA tests?
!
Data out of sync?
Server configurations out of sync?
Capacity vs. current load?
Deployment script?
32. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
How will we know when
something is wrong in
production?
!
How long will it take to
resolve the issue?
Check in
Trigger
33. We aim to reduce fundamental surprise in
every release, and we optimize for detecting
and recovering from failures quickly.
Sarabbit on Flickr:
34. “Quality is not just testing pre-release.
It also includes our adaptability and
response time.”
- Jeff Sussna at ALM Forum, 2014
35. Pre-production validation
Code deployed to de-pooled application (web) servers
touching prod services and databases.
Smoke tests
Integration tests
Functional tests
User-Acceptance (ad hoc, targeted)
36. Production validation
Exactly the same server configs, services and data as
pre-prod, but this is where we introduce application
code to live traffic.
Smoke tests (esp. over public hostnames)
User-Acceptance testing behind config flags
Gratuitous monitoring
Customer support and forums
40. Check in
Trigger
Feedback Trigger
Feedback Approval
Approval
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
Feedback
41. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
42. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
Approval
Approval
Feedback
Feedback
Feedback
43. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
Approval
Approval
Feedback
Feedback
Feedback Approval
Feedback Approval
Feedback
47. The Release Hurdle
2-3 weeks of code changes per release
Highly-orchestrated releases (6+ hours)
^ Same for rollbacks
Scheduled downtime
Long, stressful days
Slow iterations
49. First, Principles
Enable rapid product innovation
Resolve scaling hurdles
Reduce mean-time-to-recovery
Grow a talented and engaged engineering team
Reduce bureaucracy and process
66. $1.35 Billion Goods sold in 2013
60+ Million Unique visitors per month
!
200+ Committers, everyone deploys
http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet