0
Principles and Practices in
Continuous Deployment
Mike Brittain
Engineering Director, Etsy
@mikebrittain
July22,2014
“Continuous Deployment”
Process by which our team deploys software changes
to production services over 30 times per day.
The Release Hurdle
2-3 weeks of code changes per release
Highly-orchestrated releases (6+ hours)
^ Same for rollbacks
Sche...
pro·duc·tion [pruh-duhk-shuhn] (n)
1. This complex system of application code,
distributed services, servers, networking g...
Software for large-scale web sites has been
traditionally written by one group of people, then
released and operated by a ...
Stagnation
“…frequent and prolonged outages.”
2010 CAPACITY PLAN
First, Principles
Enable rapid product innovation
Resolve scaling hurdles
Reduce mean-time-to-recovery
Grow a talented and...
http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
In a software rele...
http://youtu.be/LdOe18KhtT4
Continuous Deployment
Continuous Delivery
~ vs ~
Frequent check-ins directly to
mainline.
Continuous Deployment Continuous Delivery
✓ ✓
Continuous Integration and
Automated tests.
Continuous Deployment Continuous Delivery
✓ ✓
Keep the build green.
We’re always ready to release.
Continuous Deployment Continuous Delivery
✓ ✓
“One button” deploys.
Continuous Deployment Continuous Delivery
✓ ✓
Business dictates when a build is
deployed.
Continuous Deployment Continuous Delivery
✓
Every passing build is deployed to
production.
Continuous Deployment Continuous Delivery
✓
All enhancements are gated by
Config Flags. (“Branch in code”)
Continuous Deployment Continuous Delivery
✓ ?
Most of the builds we deploy are
“dark” changes.
CSS rules, template copy, un-referenced code (classes,
functions, templat...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feed...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feed...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feed...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feed...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feed...
“What do you mean, ‘it’s not working in
production?’ I TESTED IT BEFORE WE
RELEASED!”
(DUN DUN DUUUUHHHNNN !!!)
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trig...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trig...
DUN DUN DUUUUHHHNNN !!!
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trig...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trig...
Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trig...
We aim to reduce fundamental surprise in
every release, and we optimize for detecting
and recovering from failures quickly...
“Quality is not just testing pre-release.
It also includes our adaptability and
response time.”
- Jeff Sussna at ALM Forum...
Pre-production validation
Code deployed to de-pooled application (web) servers
touching prod services and databases.
Smoke...
Production validation
Exactly the same server configs, services and data as
pre-prod, but this is where we introduce appli...
Monitoring
Monitoring
PHP Warnings Bug Reports and Help Requests
Deploy logs
Check in
Trigger
Feedback Trigger
Feedback Approval
Approval
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Co...
Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Test...
Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Test...
Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Test...
“Allow buttons properly to inherit color
from their parent node.”
The Release Hurdle
2-3 weeks of code changes per release
Highly-orchestrated releases (6+ hours)
^ Same for rollbacks
Sche...
@mikebrittain
Very end of 2009 Today
DEPLOYMENTSPERDAY
APPCODE
CONFIGFILES
First, Principles
Enable rapid product innovation
Resolve scaling hurdles
Reduce mean-time-to-recovery
Grow a talented and...
Admin-launch and whitelist
Ramp-up public traffic
US","region":"US","detected_currency_code":"USD","detected_language":"en-
US","detected_region":"US","accept-languages":"e...
Observed impact
Time series data for
duration of the
experiment
“Catapult”
Frank
Product Manager
“I want to find out whether
buyers will favor a single
price for the product that
includes shipping.”
https://www.etsy.com...
Eligibility requirements:
- Must be first page of visit
- Buyer & seller in same region
- etc…
Time: < 8 hours
Staff: One
!
Design, config flag (disabled), eligibility code in
controller, template code, CSS, code revi...
We do not bundle the item
price and shipping cost
together today.
!
https://www.etsy.com/shop/lucra
Ambitious Product Goal
Monolithic
Building and measuring many
things at once.
Ambitious Product Goal
Monolithic
Building and measuring many
things at once.
Iterative
One thing at a time, our design
go...
Time: < 8 hours
Staff: One
!
Design, config flag (disabled), eligibility code in
controller, template code, CSS, code revi...
http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
“Maybe this is jus...
etsystatus.com
@mikebrittain
Very end of 2009 Today
DEPLOYMENTSPERDAY
APPCODE
CONFIGFILES
$1.35 Billion Goods sold in 2013
60+ Million Unique visitors per month
!
200+ Committers, everyone deploys
http://www.etsy...
Thank you.
Mike Brittain
Engineering Director, Etsy
@mikebrittain
July22,2014
mikebrittain.com/talks
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
Upcoming SlideShare
Loading in...5
×

Continuous Deployment at Etsy — TimesOpen NYC

6,068

Published on

Principles and practices in Continuous Deployment at Etsy. Presented by Mike Brittain at TimesOpen (nytimes) on July 22, 2014

Published in: Engineering, Technology

Transcript of "Continuous Deployment at Etsy — TimesOpen NYC"

  1. 1. Principles and Practices in Continuous Deployment Mike Brittain Engineering Director, Etsy @mikebrittain July22,2014
  2. 2. “Continuous Deployment” Process by which our team deploys software changes to production services over 30 times per day.
  3. 3. The Release Hurdle 2-3 weeks of code changes per release Highly-orchestrated releases (6+ hours) ^ Same for rollbacks Scheduled downtime Long, stressful days Slow iterations
  4. 4. pro·duc·tion [pruh-duhk-shuhn] (n) 1. This complex system of application code, distributed services, servers, networking gear, etc., upon which we’re going to try to carefully apply a complicated set of changes and hope that nothing goes wrong. Cross your fingers… here goes.
  5. 5. Software for large-scale web sites has been traditionally written by one group of people, then released and operated by a different group. These two groups have very different levels of visibility into how the software works.
  6. 6. Stagnation
  7. 7. “…frequent and prolonged outages.” 2010 CAPACITY PLAN
  8. 8. First, Principles Enable rapid product innovation Resolve scaling hurdles Reduce mean-time-to-recovery Grow a talented and engaged engineering team Minimal bureaucracy and process
  9. 9. http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ In a software release process Fail Fast means releasing undeployed code as fast as possible, instead of waiting for a weekly release to break.
  10. 10. http://youtu.be/LdOe18KhtT4
  11. 11. Continuous Deployment Continuous Delivery ~ vs ~
  12. 12. Frequent check-ins directly to mainline. Continuous Deployment Continuous Delivery ✓ ✓
  13. 13. Continuous Integration and Automated tests. Continuous Deployment Continuous Delivery ✓ ✓
  14. 14. Keep the build green. We’re always ready to release. Continuous Deployment Continuous Delivery ✓ ✓
  15. 15. “One button” deploys. Continuous Deployment Continuous Delivery ✓ ✓
  16. 16. Business dictates when a build is deployed. Continuous Deployment Continuous Delivery ✓
  17. 17. Every passing build is deployed to production. Continuous Deployment Continuous Delivery ✓
  18. 18. All enhancements are gated by Config Flags. (“Branch in code”) Continuous Deployment Continuous Delivery ✓ ?
  19. 19. Most of the builds we deploy are “dark” changes. CSS rules, template copy, un-referenced code (classes, functions, templates), code paths behind disabled flags, etc.
  20. 20. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Source: http://en.wikipedia.org/wiki/Continuous_delivery Continuous Delivery release pipeline
  21. 21. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Source: http://en.wikipedia.org/wiki/Continuous_delivery Continuous Delivery release pipeline
  22. 22. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Source: http://en.wikipedia.org/wiki/Continuous_delivery Continuous Delivery release pipeline
  23. 23. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Continuous Delivery release pipeline Dev / Integration Staging Production
  24. 24. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Continuous Delivery release pipeline Dev / Integration Staging Production Assumptions: Deploy process is infallible. Staging is a perfect reflection of Production, with respect to hardware, configurations, data, overall load, capacity, etc.
  25. 25. “What do you mean, ‘it’s not working in production?’ I TESTED IT BEFORE WE RELEASED!” (DUN DUN DUUUUHHHNNN !!!)
  26. 26. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Dev / Integration Staging Production Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval
  27. 27. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Dev / Integration Staging Production Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback
  28. 28. DUN DUN DUUUUHHHNNN !!!
  29. 29. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback "Because you’re integrating so frequently, there is significantly less back- tracking to discover where things went wrong , so you can spend more time building features.” ! —ThoughtWorks ! ! http://www.thoughtworks.com/continuous-integration Dev / Integration Staging Production
  30. 30. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback Dev / Integration Staging Production Where’s the bug? ! In one of the numerous check-ins? Missing unit tests? Missing automated UA tests? Missing manual UA tests? ! Data out of sync? Server configurations out of sync? Capacity vs. current load? Deployment script?
  31. 31. Dev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Release Check in Trigger Trigger Approval Continuous Delivery release pipeline Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Check in Trigger Trigger Approval Approval Check in Trigger Trigger Approval Feedback Dev / Integration Staging Production How will we know when something is wrong in production? ! How long will it take to resolve the issue? Check in Trigger
  32. 32. We aim to reduce fundamental surprise in every release, and we optimize for detecting and recovering from failures quickly. Sarabbit on Flickr:
  33. 33. “Quality is not just testing pre-release. It also includes our adaptability and response time.” - Jeff Sussna at ALM Forum, 2014
  34. 34. Pre-production validation Code deployed to de-pooled application (web) servers touching prod services and databases. Smoke tests Integration tests Functional tests User-Acceptance (ad hoc, targeted)
  35. 35. Production validation Exactly the same server configs, services and data as pre-prod, but this is where we introduce application code to live traffic. Smoke tests (esp. over public hostnames) User-Acceptance testing behind config flags Gratuitous monitoring Customer support and forums
  36. 36. Monitoring
  37. 37. Monitoring PHP Warnings Bug Reports and Help Requests
  38. 38. Deploy logs
  39. 39. Check in Trigger Feedback Trigger Feedback Approval Approval Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline Feedback
  40. 40. Dev Pre-Production (“Princess”) Production Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline CI
  41. 41. Dev Pre-Production (“Princess”) Production Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline CI Approval Approval Feedback Feedback Feedback
  42. 42. Dev Pre-Production (“Princess”) Production Check in Trigger Feedback Trigger Feedback Approval ApprovalFeedback Smoke Tests User Acceptance! Tests ReleaseDev Team Version Control Build & Unit Tests Automated Acceptance Tests User Acceptance Tests Deploy (Prod) Monitoring and Automated Alerts Continuous Deployment release pipeline CI Approval Approval Feedback Feedback Feedback Approval Feedback Approval Feedback
  43. 43. “Allow buttons properly to inherit color from their parent node.”
  44. 44. The Release Hurdle 2-3 weeks of code changes per release Highly-orchestrated releases (6+ hours) ^ Same for rollbacks Scheduled downtime Long, stressful days Slow iterations
  45. 45. @mikebrittain Very end of 2009 Today DEPLOYMENTSPERDAY APPCODE CONFIGFILES
  46. 46. First, Principles Enable rapid product innovation Resolve scaling hurdles Reduce mean-time-to-recovery Grow a talented and engaged engineering team Reduce bureaucracy and process
  47. 47. Admin-launch and whitelist Ramp-up public traffic
  48. 48. US","region":"US","detected_currency_code":"USD","detected_language":"en- US","detected_region":"US","accept-languages":"en-US","cdn- provider":"","isMobileDevice":"0","isMobileSupported":"0","isMobileRequestIgnoreCookie":"0" ,"isTabletSupported":"0","isTouch":"0","isEtsyApp":"0","isPreviewRequest":"0","isChromeInst antRequest":"0","isMozPrefetchRequest":"0","listing_ids": [104073511,130604774,159651433,155451607,160523743,124025232,95186610,82967340,114692884,11 4767467,117266897,157579748],"scheduled_modules_content_ids": [10808052776,10256029946],"primary_event":"1",".event_source":"web",".event_logger":"fronte nd","php_ab_test_names":"translation_profiler.profiling;translation_profiler.logging;transl ation_profiler.backend_event_logging;footer_redesign_20131201;international.languages.el;in ternational.languages.ja;international.languages.no;international.languages.pl;internationa l.languages.ro;international.languages.tr;simplified_locale_experience;full_site_ssl;admin_ toolbar;enabled_locale_subdirectories;affiliates.publishing.user_publishers;buyer_invites_r ecipients;home_improvement;home_improvement.new_homepage;authoritative_items;refactored_foo ter;conversations.rejuvination;contextual_homepage_recs.global;css_from_www;shrinkray.css;c srf_nonce_refactor.allow_colon;csrf_nonce_refactor.reverse_order;csrf_nonce_refactor.no_enc Analytics connected to config names
  49. 49. Observed impact Time series data for duration of the experiment “Catapult”
  50. 50. Frank Product Manager
  51. 51. “I want to find out whether buyers will favor a single price for the product that includes shipping.” https://www.etsy.com/shop/lucra
  52. 52. Eligibility requirements: - Must be first page of visit - Buyer & seller in same region - etc…
  53. 53. Time: < 8 hours Staff: One ! Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled.
  54. 54. We do not bundle the item price and shipping cost together today. ! https://www.etsy.com/shop/lucra
  55. 55. Ambitious Product Goal Monolithic Building and measuring many things at once.
  56. 56. Ambitious Product Goal Monolithic Building and measuring many things at once. Iterative One thing at a time, our design goal is always in sight.
  57. 57. Time: < 8 hours Staff: One ! Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled. Deployed Deployed
  58. 58. http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ “Maybe this is just viable for a single developer … your site will be down. A lot.” ~ Informed reader
  59. 59. etsystatus.com
  60. 60. @mikebrittain Very end of 2009 Today DEPLOYMENTSPERDAY APPCODE CONFIGFILES
  61. 61. $1.35 Billion Goods sold in 2013 60+ Million Unique visitors per month ! 200+ Committers, everyone deploys http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet
  62. 62. Thank you. Mike Brittain Engineering Director, Etsy @mikebrittain July22,2014 mikebrittain.com/talks
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×